CN111782386A - Historical service flow-based database scheduling method and system - Google Patents
Historical service flow-based database scheduling method and system Download PDFInfo
- Publication number
- CN111782386A CN111782386A CN202010187678.0A CN202010187678A CN111782386A CN 111782386 A CN111782386 A CN 111782386A CN 202010187678 A CN202010187678 A CN 202010187678A CN 111782386 A CN111782386 A CN 111782386A
- Authority
- CN
- China
- Prior art keywords
- database
- server
- state data
- scheduling
- time period
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 230000008569 process Effects 0.000 claims description 9
- 238000013461 design Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000010076 replication Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a database scheduling method and a system based on historical service flow, which comprises the following steps: acquiring state data of a database called and state data of a server running the database in a first time period; calculating the growth speed of the first state data in the first time period according to the called state data of the database in the first time period; predicting the growth speed of second state data called by the database after the current time according to the growth speed of the first state data; calculating to obtain state data of a server required for operating the corresponding database after the current time according to the state data of the server of the currently-operated database in the first time period and the second state data growth speed, and judging whether the server of the currently-operated database is matched with the database calling requirement or not; if not, the server running the database is adjusted. And calculating the pressure to be borne by the future database by collecting the state data of the server and the database, and further performing service expansion in advance.
Description
Technical Field
The invention relates to the field of database scheduling, in particular to a database scheduling method and a database scheduling system based on historical service flow.
Background
With the rapid development of services, the traditional service architecture is difficult to satisfy the explosive growth of data and service traffic. The cloud computing and Docker container technology is provided with a new solution, deployment environments are packaged and standardized, when business flow and service load are increased suddenly, rapid deployment of services on a plurality of servers is realized according to existing algorithms and rules, when the business flow and the service load are reduced, redundant services can be gradually recovered, and the whole process is unaware of the business and is quick to respond, so that the method is a universal flexible scheduling method in the current industry and is a problem to be solved by a Kubelets container arrangement system.
In the process of implementing the invention, the applicant finds that at least the following problems exist in the prior art:
the above method, which is only applicable to container environment, cannot be performed when a container or a component on which the arrangement system depends fails; in addition, the method is only suitable for stateless services (such as WEB services), and for stateful servers (such as databases), because data synchronization and copying are required during scheduling, rapid scheduling cannot be realized.
Disclosure of Invention
The embodiment of the invention provides a database scheduling method and a system based on historical service flow, aiming at stateful services, the invention calculates the pressure to be borne by a future database by acquiring historical state data of various indexes such as a server, service historical database service and the like, analyzing the historical state data of the indexes, and dynamically adjusting the server according to the requirement, thereby realizing the normal calling of the database and the reasonable utilization of the server.
To achieve the above object, in one aspect, an embodiment of the present invention provides a database scheduling method based on historical service traffic, including:
acquiring and storing state data of database calls in a first time period, wherein the first time period is before the current time, and the number of the database calls in the first time period reaches a peak value; when the database is called, state data of a server which operates the database is obtained;
calculating the increasing speed of the first state data in the first time period according to the called state data of the database in the first time period; predicting the growth speed of second state data called by the database after the current time according to the growth speed of the first state data; calculating to obtain the state data of the server required for operating the corresponding database after the current time according to the state data of the server of the currently operated database in the first time period and the speed increase of the second state data;
judging whether a server of the currently running database is matched with the database calling requirement or not according to the state data of the server required for running the corresponding database after the current time; and if the server running the database does not match the database calling requirement, adjusting the server running the database.
In another aspect, an embodiment of the present invention provides a database scheduling system based on historical service traffic, including:
the historical service flow acquiring unit is used for acquiring and storing state data of the database called in a first time period, wherein the first time period is before the current time, and the number of the called databases in the first time period reaches a peak value; when the database is called, state data of a server which operates the database is obtained;
the prediction unit is used for calculating the growth speed of the first state data in the first time period according to the called state data of the database in the first time period; predicting the growth speed of second state data called by the database after the current time according to the growth speed of the first state data; calculating to obtain state data of a server required for operating a corresponding database after the current time according to the state data of the server of the currently operating database in the first time period and the second state data growth speed;
the judging unit is used for judging whether the server of the currently running database is matched with the database calling requirement or not according to the state data of the server required for running the corresponding database after the current time;
and the adjusting unit is used for adjusting the server for operating the database if the server for operating the database is not matched with the database calling requirement.
The technical scheme has the following beneficial effects: aiming at the stateful service, the invention analyzes the historical state data of various indexes such as a server, service history database service and the like by collecting the historical state data of the indexes, calculates the pressure to be borne by the future database, and dynamically adjusts the server according to the requirement, thereby realizing the normal calling of the database and the reasonable utilization of the server.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the technical solutions in the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a database scheduling method based on historical traffic flow according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a database scheduling method system based on historical service traffic according to an embodiment of the present invention;
fig. 3 is a flow chart of database scheduling based on historical traffic flow according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.
As shown in fig. 1, in combination with the embodiment of the present invention, a database scheduling method based on historical service traffic is provided, which includes:
s101: acquiring and storing state data of database calls in a first time period, wherein the first time period is before the current time, and the number of the database calls in the first time period reaches a peak value; when the database is called, state data of a server which operates the database is obtained; the state data of the server of the live database comprises: available memory allowance, available storage space allowance, available CPU idle rate, disk type and network card type;
s102: calculating the growth speed of the first state data in the first time period according to the called state data of the database in the first time period; predicting the growth speed of second state data called by the database after the current time according to the growth speed of the first state data; according to the state data of the server of the currently running database in the first time period and the second state data increasing speed, calculating to obtain the state data of the server required for running the corresponding database after the current time;
s103: judging whether the server of the currently running database is matched with the database calling requirement or not according to the state data of the server required for running the corresponding database after the current time;
s104: and if the server running the database does not match the database calling requirement, adjusting the server running the database.
Preferably, the state data called by the database includes: query rate per second QPS, occupied memory size, occupied storage space size, CPU utilization rate and system throughput TPS;
calculating the increasing speed of the first state data in the first time period according to the called state data of the database in the first time period, which specifically comprises the following steps:
and calculating the growth speed of the first state data in the first time period by a weighted moving average method according to the query rate per second QPS, the occupied memory size, the occupied memory space size, the CPU utilization rate and the system throughput TPS.
Preferably, it further comprises S105:
acquiring and storing rule information of a database scheduling server in a distributed manner, wherein the rule information of the database scheduling server comprises: a list of server rooms and all servers, whether the database has to be deployed to certain rooms, whether the database needs to be deployed in multiple rooms, and a list of databases.
Preferably, step 103 specifically includes:
s1031: when the state data of the server required for operating the corresponding database after the current time is smaller than a preset first threshold and larger than a preset second threshold, judging that the server of the currently operating database is matched with the database calling requirement, wherein the first threshold is larger than the second threshold; otherwise, judging that the server of the currently running database is not matched with the database calling requirement.
Preferably, step 104 specifically includes:
s1041: when the state data of the server required for operating the corresponding database after the current time is not less than a preset first threshold value, judging that the server of the currently operating database does not meet the database calling requirement, adding other servers for the database according to the rule information of the database scheduling server, and synchronizing the database into the added servers; the state data of the server required for operating the corresponding database comprises: the required memory size, the required storage space size, the occupied CPU size and the maximum connection number;
s1042: and when the state data of the server required for operating the corresponding database after the current time is not greater than a preset second threshold value, judging that the server of the currently operated database is redundant relative to the database calling requirement, and reducing the servers of the operated database.
Preferably, in step 1041: the adding other servers to the database according to the rule information of the database scheduling server, and synchronizing the database to the added servers specifically includes:
and acquiring a list of the current available servers from the lists of all the servers, acquiring state data of the current available servers, and increasing the current available servers conforming to the operation database into the servers operating the database according to the rule information of the database scheduling server.
Preferably, the method further comprises the following steps: and when receiving an instruction of scheduling the database, generating a corresponding scheduling task, and scheduling the corresponding database through the scheduling task. As shown in fig. 2, in combination with the embodiment of the present invention, there is also provided a database scheduling system based on historical service traffic, including:
a historical service traffic obtaining unit 21, configured to obtain and store state data of a database that is called in a first time period, where the first time period is before a current time, and a number of called databases in the first time period reaches a peak value; when the database is called, state data of a server which operates the database is obtained;
the prediction unit 22 is used for calculating the first state data growth speed in the first time period according to the state data of the database called in the first time period; predicting the growth speed of second state data called by the database after the current time according to the growth speed of the first state data; calculating to obtain state data of a server required for operating a corresponding database after the current time according to the state data of the server of the currently operating database in the first time period and the second state data growth speed;
the judging unit 23 is configured to judge whether the server currently running the database matches the database call requirement according to the state data of the server needed to run the corresponding database after the current time;
the adjusting unit 24 adjusts the server running the database if the server currently running the database does not match the database call requirement.
Preferably, the state data called by the database includes: query rate per second QPS, occupied memory size, occupied storage space size, CPU utilization rate and system throughput TPS;
the prediction unit 22 is specifically configured to:
and calculating the growth speed of the first state data in the first time period by a weighted moving average method according to the query rate per second QPS, the occupied memory size, the occupied memory space size, the CPU utilization rate and the system throughput TPS.
Preferably, the system further comprises a service rule obtaining unit 25, configured to obtain and store rule information of the database scheduling server in a distributed manner, where the rule information of the database scheduling server includes: a list of server rooms and all servers, whether a database service has to be deployed to certain rooms, whether a database service requires multi-room deployment, a list of database services.
Preferably, the determining unit 23 is specifically configured to:
when the state data of the server required for operating the corresponding database after the current time is smaller than a preset first threshold and larger than a preset second threshold, judging that the server of the currently operating database is matched with the database calling requirement, wherein the first threshold is larger than the second threshold; otherwise, judging that the server of the currently running database is not matched with the database calling requirement.
Preferably, the adjusting unit 24 specifically includes:
a first adjusting subunit 241, configured to, when state data of a server required to run a corresponding database after the current time is not less than a preset first threshold, determine that the server currently running the database does not meet a database call requirement, add another server to the database according to rule information of a database scheduling server, and synchronize the database into the added server; the state data of the server required for operating the corresponding database comprises: the required memory size, the required storage space size, the occupied CPU size and the maximum connection number;
and a second adjusting subunit 241, configured to, when the state data of the server required for running the corresponding database after the current time is not greater than a preset second threshold, determine that the server of the currently running database is redundant with respect to the database call requirement, and reduce the servers of the running database.
Preferably, the first adjusting subunit 241 is specifically configured to:
and in the process of adding other servers to the database according to the rule information of the database scheduling server, acquiring a list of the current available server from the lists of all the servers, acquiring the state data of the current available server, and adding the current available server conforming to the operation of the database into the service for operating the database according to the rule information of the database scheduling server.
Preferably, when receiving an instruction to schedule the database, a corresponding scheduling task is generated, and the corresponding database is scheduled by the scheduling task.
The invention has the following beneficial effects: aiming at the stateful service, the invention analyzes the historical state data of various indexes such as a server, a business historical database service and the like by collecting the historical state data of the indexes, calculates the pressure to be borne by the future database, and expands the capacity of the server according to the requirement, thereby realizing the normal calling of the database.
As shown in fig. 3, the following describes the above technical solution of the embodiment of the present invention in detail with reference to specific application examples, and reference may be made to the related description above for technical details that are not described in the implementation process.
Aiming at the stateful service, the invention adopts the analysis of historical access data based on a database and dynamically sends out a scheduling instruction, and is a dynamic scheduling method aiming at the stateful service.
(1) First, there is a set of information collection system, and the collection system is used to collect data in a first time period, where the collected data includes various states of a server (traffic history) and various states of a database service, where the first time period is before a current time, and the number of database calls in the first time period reaches a peak value. The various states of the server include, for example: available memory (i.e., referring to the size of the remaining memory), storage space (i.e., the size of the available storage space), CPU (i.e., "CPU utilization," "CPU idleness"), disk type, network card, and so forth. The disk types are storage types, and the disk types are different from SAS, SSD, NVME, etc., and each type has different performance (such as IOPS, throughput, etc.). The network card types include a gigabit network card, a tera network card, and the like (gigabit means a network card rate, and the gigabit is 1000 Mb/second). Judging which type of resources are more suitable for being scheduled according to various performance indexes of the service, for example, the service with high QPS/TPS is more likely to be scheduled to the storage of SSD or NVME type, so that the performance is better and the service response speed is higher; the high-Read-Bytes/Write-Bytes service is more likely to be dispatched to the ten-gigabit network card because the throughput of the network card is high. Various states of the database service include, for example: query rate per second QPS (Queries-per-second), memory usage, storage space occupation, central processing unit CPU usage and the like, and database services comprise database services such as MySQL, Redis, MongoDB and the like; a database process deployed on a server is referred to as a database instance, and multiple database instances (multiple database processes) may be deployed on a server. Sometimes, a database is deployed on multiple servers, and multiple database instances (generally, the same port number, with 1 Master and multiple slave) with data replication relationship are called a set of database cluster.
(2) And the data storage center adopts Etcd (which is a distributed storage service) as storage, and stores the state information collected in the step (1) and the rule information of the database scheduling server. The rule information of the database scheduling server comprises: for example, a server room list (including a list of all rooms and a list of all servers), a database master-slave must be deployed to some rooms, whether multi-room deployment is needed, a database service list (including a list of database use port numbers, the number of database instances of each database cluster, and the like), configuration information of the current deployment of the database (information such as the memory size, the storage space size, the maximum connection number, and the like of the database instances), and the like;
(3) and (3) analyzing the database service data collected in the step (1) and stored in the step (2), namely predicting the access growth in a future period after the current period according to the database pressure and resource requirements in the access peak period in the recent period (in the first period). Calculating the growth speed of the first state data in the first time period according to the called state data of the database in the first time period; predicting the growth speed of second state data called by the database after the current time according to the growth speed of the first state data; and calculating the state data of the server required for operating the corresponding database after the current time according to the state data of the server of the currently operated database in the first time period and the second state data growth speed. The database pressure specifically refers to database performance indexes, namely, QPS, system throughput TPS, and the like; the resource requirement specifically refers to the requirements of deployment environments and resources, such as tera network cards, SSD solid state drives, and the like. The database needs how many database instances, what configuration to carry these database instance stresses, in the current or future period of time, are predicted from changes in the performance metrics of the service. Examples are: predicting whether the current deployment and capacity of the database can meet the future access pressure (performance index); such as: the QPS (requests per second) index of the database cluster with port number 3303 in this week is gradually increased from 5000 to 7500, and according to this information, the QPS of the database cluster is predicted to continue to increase by 50% in the future week, so that 1 slave is added to the database cluster in advance (before the end of the future week) to cope with the possible future increase in access amount.
(4) And (3) a server state analysis program for analyzing the server data collected in the step (1) and stored in the step (2), and acquiring a currently available server list and details of available resources of the server from the server room list, wherein: details of the resources available to the server include: the residual memory of a certain server, the residual storage space, the available bandwidth of the tera network card and the like; specifically, the remaining available resources of the server (for example, the size of the remaining memory, the size of the remaining storage space, and the like) and the configuration needs of the database instance currently needing to be scheduled (how much memory, how much storage space, and the like) are analyzed, and a "database instance configuration need needing to be scheduled" that can be satisfied by all the servers (remaining resources) is obtained.
(5) The scheduling task generator is used for judging whether the server of the currently running database is matched with the database call according to the state data of the server required by running the corresponding database after the current time; and if the server running the database does not match the database calling requirement, adjusting the server running the database. And when the state data of the server required for operating the corresponding database after the current time is smaller than a preset first threshold and larger than a preset second threshold, judging that the server of the currently operated database is matched with the database calling requirement, wherein the first threshold is larger than the second threshold. Specifically, according to the pressure increase condition of the database calculated in the step (3) in a future period of time, comparing with the obtained current database deployment condition in the step (2), if the current database is found to be incapable of bearing the access pressure of the database in the future period of time, distributing one or more available servers from an available server list according to the server resource information obtained in the step (4), generating a service scheduling task, and finally performing the task execution operation through the step (6); wherein, allocating one or more available servers specifically refers to: according to the acquired 'server available resource list', and according to the configuration requirement of the database instance needing to be scheduled currently, a server list of the database instance available deployment is acquired. Such as: a certain server has 20G of residual memory, but a certain database instance needs 25G of memory, although the server has available memory, the configuration requirement cannot be met, and the server is excluded. The operations of synchronization and copying are realized quickly during scheduling through a database internal data full synchronization mechanism (such as database services of MongoDB, Redis and the like) and a database external data synchronization tool (such as rsync).
Otherwise, judging that the server of the currently running database is not matched with the database calling requirement. Scheduling in advance, when the state data of the server required for operating the corresponding database after the current time is not less than a preset first threshold value, judging that the server of the currently operating database does not meet the database calling requirement, adding other servers for the database according to the rule information of the database scheduling server, and synchronizing the database into the added servers; the state data of the server required for operating the corresponding database comprises: the required memory size, the required storage space size, the occupied CPU size and the maximum connection number. Examples are: according to the fluctuation condition of each performance index of database service, if the database pressure (indexes such as QPS, TPS and the like) is increased by 1 time in 10 minutes, the scheduling is initiated immediately; if the pressure increases 0.5 times in 1 day, scheduling is initiated before the point in time when traffic is expected to increase 1 times (1 day is expected). After the current time, when the increasing speed of the database service is a preset multiple n relative to the increasing speed of the database service at a certain previous time point, and the state data of the server required by the corresponding database to operate is not less than a preset first threshold, adding other servers for the database at the time point according to the scheduling rule of the database, wherein n is greater than 1. And during scheduling, acquiring a list of the current available servers from the lists of all the servers, acquiring state data of the current available servers, and increasing the current available servers conforming to the operation of the database into the servers operating the database according to the rule information of the database scheduling server.
In addition, the advance scheduling further includes: after the current time, when the service growth speed of the database is a preset multiple n relative to the growth speed of the database at a previous time point, and when the state data of the server required by running the corresponding database after the current time is not greater than a preset second threshold, the server of the currently running database is judged to be redundant relative to the database call demand, and then the servers of the running database are reduced, wherein n is less than 1.
(6) And (5) a scheduling task executor which schedules the corresponding service to the corresponding server according to the specific scheduling task generated in the step (5), namely generates a corresponding scheduling task when receiving an instruction of a scheduling database, and schedules corresponding data through the scheduling task. The specific scheduling task refers to: scheduling a task is a real operation to be performed, such as: the database with port number 3303 requires 20G of memory and 200G of storage space, and is scheduled to the server with IP of 10.0.0.1, and the remaining memory of the server has 50G and 1T of storage space.
Compared with the prior art, the invention has the following advantages:
first, the entire system does not rely on containers or other orchestration tools;
secondly, the scheduling system is different from other scheduling systems existing in the industry, and is specially designed for stateful services such as a database; the method is suitable for stateless service (the stateless service is characterized in that the request does not depend on the previous request information, is light in weight and high in starting speed, such as WEB service, and the stateful service is characterized in that the request depends on the previous request information, is heavy in service, and needs specific data storage media (such as a disk, a memory and the like), such as database service).
Thirdly, the future access pressure can be predicted according to the historical access condition of the database, scheduling is initiated in advance, and the situation that the scheduling cannot be rapidly realized when the access flow is sudden is prevented. Therefore, whether the current service can bear the access volume in the current or future period of time or not is judged according to the dimensions of the increment/decrement, the increment/decrement speed, the increment/decrement rule and the like of the QPT and the TPS, and actions such as capacity expansion, capacity contraction and the like are further triggered. The data (such as the service flow) based on the scheduling can only be the current real-time data, the more intelligent scheduling can not be performed on the historical service flow, and the advance scheduling can not be performed on the service with the regularly changed flow.
In summary, the invention analyzes the historical data of the indexes by collecting various indexes such as a server, a business historical database service and the like aiming at the stateful service, calculates the pressure which is possibly born by the future database, dynamically sends out a scheduling instruction, and further performs service expansion in advance, and the whole system does not depend on a container or an arrangement system, namely the method of the invention can realize index collection-containerization-database-flexible scheduling.
It should be understood that the specific order or hierarchy of steps in the processes disclosed is an example of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged without departing from the scope of the present disclosure. The accompanying method claims present elements of the various steps in a sample order, and are not intended to be limited to the specific order or hierarchy presented.
In the foregoing detailed description, various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments of the subject matter require more features than are expressly recited in each claim. Rather, as the following claims reflect, invention lies in less than all features of a single disclosed embodiment. Thus, the following claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. To those skilled in the art; various modifications to these embodiments will be readily apparent, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
What has been described above includes examples of one or more embodiments. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the aforementioned embodiments, but one of ordinary skill in the art may recognize that many further combinations and permutations of various embodiments are possible. Accordingly, the embodiments described herein are intended to embrace all such alterations, modifications and variations that fall within the scope of the appended claims. Furthermore, to the extent that the term "includes" is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term "comprising" as "comprising" is interpreted when employed as a transitional word in a claim. Furthermore, any use of the term "or" in the specification of the claims is intended to mean a "non-exclusive or".
Those of skill in the art will further appreciate that the various illustrative logical blocks, units, and steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate the interchangeability of hardware and software, various illustrative components, elements, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design requirements of the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the embodiments of the present invention.
The various illustrative logical blocks, or elements, described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor, an Application Specific Integrated Circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a digital signal processor core, or any other similar configuration.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may be stored in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. For example, a storage medium may be coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC, which may be located in a user terminal. In the alternative, the processor and the storage medium may reside in different components in a user terminal.
In one or more exemplary designs, the functions described above in connection with the embodiments of the invention may be implemented in hardware, software, firmware, or any combination of the three. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media that facilitate transfer of a computer program from one place to another. Storage media may be any available media that can be accessed by a general purpose or special purpose computer. For example, such computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store program code in the form of instructions or data structures and which can be read by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Additionally, any connection is properly termed a computer-readable medium, and, thus, is included if the software is transmitted from a website, server, or other remote source via a coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL), or wirelessly, e.g., infrared, radio, and microwave. Such discs (disk) and disks (disc) include compact disks, laser disks, optical disks, DVDs, floppy disks and blu-ray disks, where disks usually reproduce data magnetically, while disks usually reproduce data optically with lasers. Combinations of the above may also be included in the computer-readable medium.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are merely exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (12)
1. A database scheduling method based on historical service flow is characterized by comprising the following steps:
acquiring and storing state data of database calls in a first time period, wherein the first time period is before the current time, and the number of the database calls in the first time period reaches a peak value; when the database is called, state data of a server which operates the database is obtained;
calculating the growth speed of the first state data in the first time period according to the called state data of the database in the first time period; predicting the growth speed of second state data called by the database after the current time according to the growth speed of the first state data; calculating to obtain state data of a server required for operating a corresponding database after the current time according to the state data of the server of the currently operating database in the first time period and the second state data growth speed;
judging whether the server of the currently running database is matched with the database calling requirement or not according to the state data of the server required for running the corresponding database after the current time;
and if the server running the database does not match the database calling requirement, adjusting the server running the database.
2. The database scheduling method based on historical service traffic as claimed in claim 1, wherein the status data called by the database comprises: query rate per second QPS, occupied memory size, occupied storage space size, CPU utilization rate and system throughput TPS;
calculating the increasing speed of the first state data in the first time period according to the called state data of the database in the first time period, which specifically comprises the following steps:
and calculating the growth speed of the first state data in the first time period by a weighted moving average method according to the query rate per second QPS, the occupied memory size, the occupied memory space size, the CPU utilization rate and the system throughput TPS.
3. The database scheduling method based on historical traffic flow according to claim 1 or 2, further comprising:
acquiring and storing rule information of a database scheduling server in a distributed manner, wherein the rule information of the database scheduling server comprises: a list of server rooms and all servers, whether the database has to be deployed to certain rooms, whether the database needs multi-room deployment, and a list of databases.
4. The database scheduling method based on historical service traffic according to claim 3, wherein the step of judging whether the server currently running the database matches the database call requirement according to the state data of the server needed to run the corresponding database after the current time specifically comprises:
when the state data of the server required for operating the corresponding database after the current time is smaller than a preset first threshold and larger than a preset second threshold, judging that the server of the currently operating database is matched with the database calling requirement, wherein the first threshold is larger than the second threshold; otherwise, judging that the server of the currently running database is not matched with the database calling requirement.
5. The database scheduling method based on historical service traffic according to claim 4, wherein if the server currently running the database does not match the database call demand, adjusting the server running the database specifically includes:
when the state data of the server required for operating the corresponding database after the current time is not less than a preset first threshold value, judging that the server of the currently operating database does not meet the database calling requirement, adding other servers for the database according to the rule information of the database scheduling server, and synchronizing the database into the added servers; the state data of the server required for operating the corresponding database comprises: the required memory size, the required storage space size, the occupied CPU size and the maximum connection number;
and when the state data of the server required for operating the corresponding database after the current time is not greater than a preset second threshold value, judging that the server of the currently operated database is redundant relative to the database calling requirement, and reducing the servers of the operated database.
6. The database scheduling method based on historical service traffic according to claim 5, wherein the adding other servers to the database according to the rule information of the database scheduling server and synchronizing the database to the added servers specifically includes:
and acquiring a list of the current available servers from the lists of all the servers, acquiring state data of the current available servers, and increasing the current available servers conforming to the operation database into the servers operating the database according to the rule information of the database scheduling server.
7. A database scheduling system based on historical traffic, comprising:
the historical service flow acquiring unit is used for acquiring and storing state data of the database called in a first time period, wherein the first time period is before the current time, and the number of the called databases in the first time period reaches a peak value; when the database is called, state data of a server which operates the database is obtained;
the prediction unit is used for calculating the growth speed of the first state data in the first time period according to the called state data of the database in the first time period; predicting the growth speed of second state data called by the database after the current time according to the growth speed of the first state data; calculating to obtain state data of a server required for operating a corresponding database after the current time according to the state data of the server of the currently operating database in the first time period and the second state data growth speed;
the judging unit is used for judging whether the server of the currently running database is matched with the database calling requirement or not according to the state data of the server required for running the corresponding database after the current time;
and the adjusting unit is used for adjusting the server for operating the database if the server for operating the database is not matched with the database calling requirement.
8. The historical traffic-based database scheduling system according to claim 7, wherein the database-invoked state data comprises: query rate per second QPS, occupied memory size, occupied storage space size, CPU utilization rate and system throughput TPS;
the prediction unit is specifically configured to:
and calculating the growth speed of the first state data in the first time period by a weighted moving average method according to the query rate per second QPS, the occupied memory size, the occupied memory space size, the CPU utilization rate and the system throughput TPS.
9. The database scheduling system based on historical traffic flow according to claim 7 or 8, further comprising:
a service rule obtaining unit, configured to obtain rule information of a database scheduling server and store the rule information in a distributed manner, where the rule information of the database scheduling server includes: a list of server rooms and all servers, whether a database service has to be deployed to certain rooms, whether a database service requires multi-room deployment, a list of database services.
10. The database scheduling system based on historical traffic flow according to claim 9, wherein the determining unit is specifically configured to:
when the state data of the server required for operating the corresponding database after the current time is smaller than a preset first threshold and larger than a preset second threshold, judging that the server of the currently operating database is matched with the database calling requirement, wherein the first threshold is larger than the second threshold; otherwise, judging that the server of the currently running database is not matched with the database calling requirement.
11. The database scheduling system based on historical service traffic according to claim 10, wherein the adjusting unit specifically includes:
the first adjusting subunit is used for judging that the server currently running the database does not meet the database calling requirement when the state data of the server required for running the corresponding database after the current time is not smaller than a preset first threshold value, adding other servers for the database according to the rule information of the database scheduling server, and synchronizing the database into the added servers; the state data of the server required for operating the corresponding database comprises: the required memory size, the required storage space size, the occupied CPU size and the maximum connection number;
and the second adjusting subunit is used for judging that the server of the currently running database is redundant relative to the database calling requirement when the state data of the server required for running the corresponding database after the current time is not greater than a preset second threshold value, and reducing the servers of the running database.
12. The database scheduling system based on historical traffic flow according to claim 11, wherein the first adjusting subunit is specifically configured to:
in the process of adding other servers to the database according to the rule information of the database scheduling server, a list of the current available server is obtained from the lists of all the servers, the state data of the current available server is obtained, and the current available server which accords with the operation of the database is added to be the server which operates the database according to the rule information of the database scheduling server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010187678.0A CN111782386B (en) | 2020-03-17 | 2020-03-17 | Database scheduling method and system based on historical service flow |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010187678.0A CN111782386B (en) | 2020-03-17 | 2020-03-17 | Database scheduling method and system based on historical service flow |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111782386A true CN111782386A (en) | 2020-10-16 |
| CN111782386B CN111782386B (en) | 2024-07-26 |
Family
ID=72753175
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010187678.0A Active CN111782386B (en) | 2020-03-17 | 2020-03-17 | Database scheduling method and system based on historical service flow |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111782386B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112380011A (en) * | 2020-11-16 | 2021-02-19 | 中国工商银行股份有限公司 | Dynamic adjustment method and device for service capacity |
| CN112596995A (en) * | 2020-12-26 | 2021-04-02 | 中国农业银行股份有限公司 | Capacity determination method and device based on cluster architecture |
| CN112685390A (en) * | 2021-03-22 | 2021-04-20 | 阿里云计算有限公司 | Database instance management method and device and computing equipment |
| CN113448685A (en) * | 2021-06-07 | 2021-09-28 | 新浪网技术(中国)有限公司 | Pod scheduling method and system based on Kubernetes |
| CN115114106A (en) * | 2021-03-23 | 2022-09-27 | 中国移动通信集团河南有限公司 | Method, device and equipment for processing account-out task |
| CN115914403A (en) * | 2022-11-16 | 2023-04-04 | 中国工商银行股份有限公司 | Distributed service registration processing method and device |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104283946A (en) * | 2014-09-26 | 2015-01-14 | 东北大学 | Resource self-adaptive adjusting system and method of multiple virtual machines under single physical machine |
| CN109032914A (en) * | 2018-09-06 | 2018-12-18 | 掌阅科技股份有限公司 | Resource occupation data predication method, electronic equipment, storage medium |
| CN109271453A (en) * | 2018-10-22 | 2019-01-25 | 阿里巴巴集团控股有限公司 | A kind of determination method and apparatus of database volume |
| CN109271372A (en) * | 2018-09-12 | 2019-01-25 | 杭州沃趣科技股份有限公司 | A kind of automated date base management method based on state in the period and performance change |
| CN109343965A (en) * | 2018-10-31 | 2019-02-15 | 北京金山云网络技术有限公司 | Resource adjusting method, device, cloud platform and server |
| CN109446041A (en) * | 2018-09-25 | 2019-03-08 | 平安普惠企业管理有限公司 | A kind of server stress method for early warning, system and terminal device |
| CN110442453A (en) * | 2019-08-01 | 2019-11-12 | 佛山普瑞威尔科技有限公司 | A kind of automatic telescopic strategy O&M method, system and readable storage medium storing program for executing |
| CN110445650A (en) * | 2019-08-07 | 2019-11-12 | 中国联合网络通信集团有限公司 | Detect alarm method, equipment and server |
| CN110677459A (en) * | 2019-09-02 | 2020-01-10 | 金蝶软件（中国）有限公司 | Resource adjusting method and device, computer equipment and computer storage medium |
-
2020
- 2020-03-17 CN CN202010187678.0A patent/CN111782386B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104283946A (en) * | 2014-09-26 | 2015-01-14 | 东北大学 | Resource self-adaptive adjusting system and method of multiple virtual machines under single physical machine |
| CN109032914A (en) * | 2018-09-06 | 2018-12-18 | 掌阅科技股份有限公司 | Resource occupation data predication method, electronic equipment, storage medium |
| CN109271372A (en) * | 2018-09-12 | 2019-01-25 | 杭州沃趣科技股份有限公司 | A kind of automated date base management method based on state in the period and performance change |
| CN109446041A (en) * | 2018-09-25 | 2019-03-08 | 平安普惠企业管理有限公司 | A kind of server stress method for early warning, system and terminal device |
| CN109271453A (en) * | 2018-10-22 | 2019-01-25 | 阿里巴巴集团控股有限公司 | A kind of determination method and apparatus of database volume |
| CN109343965A (en) * | 2018-10-31 | 2019-02-15 | 北京金山云网络技术有限公司 | Resource adjusting method, device, cloud platform and server |
| CN110442453A (en) * | 2019-08-01 | 2019-11-12 | 佛山普瑞威尔科技有限公司 | A kind of automatic telescopic strategy O&M method, system and readable storage medium storing program for executing |
| CN110445650A (en) * | 2019-08-07 | 2019-11-12 | 中国联合网络通信集团有限公司 | Detect alarm method, equipment and server |
| CN110677459A (en) * | 2019-09-02 | 2020-01-10 | 金蝶软件（中国）有限公司 | Resource adjusting method and device, computer equipment and computer storage medium |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112380011A (en) * | 2020-11-16 | 2021-02-19 | 中国工商银行股份有限公司 | Dynamic adjustment method and device for service capacity |
| CN112380011B (en) * | 2020-11-16 | 2024-02-27 | 中国工商银行股份有限公司 | Dynamic adjustment method and device for service capacity |
| CN112596995A (en) * | 2020-12-26 | 2021-04-02 | 中国农业银行股份有限公司 | Capacity determination method and device based on cluster architecture |
| CN112685390A (en) * | 2021-03-22 | 2021-04-20 | 阿里云计算有限公司 | Database instance management method and device and computing equipment |
| CN115114106A (en) * | 2021-03-23 | 2022-09-27 | 中国移动通信集团河南有限公司 | Method, device and equipment for processing account-out task |
| CN113448685A (en) * | 2021-06-07 | 2021-09-28 | 新浪网技术(中国)有限公司 | Pod scheduling method and system based on Kubernetes |
| CN113448685B (en) * | 2021-06-07 | 2024-03-22 | 新浪技术（中国）有限公司 | Pod scheduling method and system based on Kubernetes |
| CN115914403A (en) * | 2022-11-16 | 2023-04-04 | 中国工商银行股份有限公司 | Distributed service registration processing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111782386B (en) | 2024-07-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111782386B (en) | Database scheduling method and system based on historical service flow | |
| US10715460B2 (en) | Opportunistic resource migration to optimize resource placement | |
| AU2011320763B2 (en) | System and method of active risk management to reduce job de-scheduling probability in computer clusters | |
| CN113010260B (en) | Container number elastic expansion method and container number elastic expansion method system | |
| CN109087055B (en) | Service request control method and device | |
| CN108776934A (en) | Distributed data computational methods, device, computer equipment and readable storage medium storing program for executing | |
| CN109739627B (en) | Task scheduling method, electronic device and medium | |
| CN112165436A (en) | Flow control method, device and system | |
| CN106201839B (en) | Information loading method and device for business object | |
| CN113485792B (en) | Pod scheduling method in kubernetes cluster, terminal equipment and storage medium | |
| US11765099B2 (en) | Resource allocation using distributed segment processing credits | |
| CN105450784B (en) | The device and method of message distribution consumption node into MQ | |
| KR101810180B1 (en) | Method and apparatus for distributed processing of big data based on user equipment | |
| CN116248699B (en) | Data reading method, device, equipment and storage medium in multi-copy scene | |
| CN117215493A (en) | file sharing system | |
| EP3807752A1 (en) | Volatile account identification and isolation and resource management in distributed data storage systems | |
| CN115878309A (en) | Resource allocation method, device, processing core, equipment and computer readable medium | |
| CN118396140B (en) | Distributed model training system and method | |
| US12028271B2 (en) | Prioritizing messages for server processing based on monitoring and predicting server resource utilization | |
| CN118377635B (en) | Message rapid processing method based on high concurrency scene | |
| US20230401089A1 (en) | Credit-based scheduling using load prediction | |
| CN115114309A (en) | Data merging processing method and device, electronic equipment and storage medium | |
| US11860942B1 (en) | Predictive loading and unloading of customer data in memory | |
| CN105117291A (en) | Method and system adjusting operation quantity of message push executor | |
| CN115098247A (en) | Resource allocation method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230412 Address after: Room 501-502, 5/F, Sina Headquarters Scientific Research Building, Block N-1 and N-2, Zhongguancun Software Park, Dongbei Wangxi Road, Haidian District, Beijing, 100193 Applicant after: Sina Technology (China) Co.,Ltd. Address before: 100193 7th floor, scientific research building, Sina headquarters, plot n-1, n-2, Zhongguancun Software Park, Dongbei Wangxi Road, Haidian District, Beijing, 100193 Applicant before: Sina.com Technology (China) Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |