CN117235189A - Method for improving IO performance of parallel data warehouse by using distributed virtual storage - Google Patents
Method for improving IO performance of parallel data warehouse by using distributed virtual storage Download PDFInfo
- Publication number
- CN117235189A CN117235189A CN202310162137.6A CN202310162137A CN117235189A CN 117235189 A CN117235189 A CN 117235189A CN 202310162137 A CN202310162137 A CN 202310162137A CN 117235189 A CN117235189 A CN 117235189A
- Authority
- CN
- China
- Prior art keywords
- preset
- database
- evaluation parameter
- server
- control module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 238000011156 evaluation Methods 0.000 claims abstract description 290
- 230000005540 biological transmission Effects 0.000 claims description 34
- 230000008569 process Effects 0.000 claims description 12
- 230000005484 gravity Effects 0.000 claims description 10
- 238000012937 correction Methods 0.000 claims description 6
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002028 premature Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the technical field of storage virtualization, in particular to a method for improving IO performance of a parallel data warehouse by using distributed virtual storage, which comprises the following steps: step S1, a central control module judges the running state of a database according to a calculated database IO performance evaluation parameter P; step S2, adjusting preset evaluation parameters of the database according to the byte number of the data read by the database once; step S3, the overflow data of the database is adjusted to an auxiliary database of the database according to P, and the reserved memory of the single database is adjusted according to the total amount of the newly added auxiliary database and the average evaluation parameter values of a plurality of databases in the single server; step S4, adjusting the data in the server according to the acquired average evaluation parameter value of the single server; the invention effectively improves the IO performance of the parallel data warehouse and further improves the running speed of the parallel distributed data warehouse.
Description
Technical Field
The invention relates to the technical field of storage virtualization, in particular to a method for improving IO performance of a parallel data warehouse by using distributed virtual storage.
Background
With the development and application of internet technology, a large amount of data to be stored will be generated in the application process of the internet, and in the conventional storage technology, when the data generated by the user is more, the situation of insufficient local storage equipment will be encountered.
Chinese patent publication No.: CN106708428A discloses a data virtualized storage method and device, the method comprises: acquiring a storage request sent by a terminal; obtaining data to be stored according to a storage request, and extracting a virtual storage path and a virtual storage instruction in the storage request; searching a distributed storage path corresponding to the virtual storage path, and searching a distributed storage instruction corresponding to the virtual storage instruction in a distributed instruction library; and storing the data to be stored to the distributed storage device according to the distributed storage path and the distributed storage instruction. The data virtualization storage method and the data virtualization storage device can link the terminal with the distributed storage equipment, provide a distributed storage function for the terminal and store the storage data sent by the terminal into the distributed storage equipment; it can be seen that the data virtualization storage method and device have the following problems: memory planning is not performed on the distributed storage equipment, so that IO performance of the parallel data warehouse is affected, and further the running speed of the parallel distributed data warehouse is affected.
Disclosure of Invention
Therefore, the invention provides a method for improving IO performance of a parallel data warehouse by using distributed virtual storage, which is used for solving the problems that in the prior art, memory planning is not performed on distributed storage equipment, IO performance of the parallel data warehouse is influenced, and then the running speed of the parallel distributed data warehouse is influenced.
To achieve the above object, the present invention provides a method for improving IO performance of a parallel data warehouse by using distributed virtual storage, including:
step S1, a central control module calculates a database IO performance evaluation parameter P according to the number of bytes read by a single database and the average transmission rate, and judges whether the running state of the database meets a preset standard according to the P;
step S2, the central control module adjusts the early warning level to the corresponding alarm level according to the proportion of the calculated total interrupt duration of a plurality of transmission interrupt processes occurring in the data transmission process of the database in the preset duration, and adjusts the preset evaluation parameters of the database according to the byte number of the data read by the database once;
step S3, the central control module adjusts overflow data of the database to an affiliated database of the database according to the IO performance evaluation parameter P of the database, and adjusts the proportion of reserved memory of the single database to the total memory of the server according to the total amount of the newly added affiliated database and the average evaluation parameter values of a plurality of databases in the single server;
And S4, after the comparison and the readjustment, the central control module adjusts the data in the servers according to the acquired average evaluation parameter value of the single server, the highest average evaluation parameter value and the lowest average evaluation parameter value in the plurality of servers.
Further, in the step S1, the central control module calculates the database IO performance evaluation parameters P, P according to the number of bytes read by a single database and the average transmission rate, where the calculation formula of the database IO performance evaluation parameters P, P is,wherein D is i For the average transmission rate of the database in the ith preset period, i=1, 2,3 … n is set, Z is the number of bytes read by a single database at a time, alpha is a first evaluation correction coefficient, alpha=0.05 s is set, beta is a second evaluation correction coefficient, and beta=50 kb is set -1 。
Further, the central control module determines whether the running state of the database accords with the judging mode of the preset standard according to the IO performance evaluation parameter P of the database, wherein,
the first judging mode is that the central control module judges that the running state of the database meets the preset standard and controls the database to run with the current running parameters;
the second judging mode is that the central control module judges that the network environment of the server corresponding to the database does not accord with a preset standard, invokes the total interrupt duration of a plurality of transmission interrupt processes of the database in the data transmission process to calculate the proportion of the total interrupt duration in the preset duration, and judges whether the network environment of the server accords with the preset standard or not according to the calculation result;
The third judging mode is that the central control module judges that the running state of the database does not accord with a preset standard, and the central control module obtains the total data quantity in the database and adjusts the total data quantity of the database to a corresponding value according to the difference value between the evaluation parameter P and the preset standard evaluation parameter;
the fourth judging mode is that the central control module judges that the IO basic performance of the database does not accord with a preset standard, and the central control module sends out alarm information aiming at replacing corresponding hardware and reduces the preset evaluation parameter to a corresponding value;
the first judging mode meets the condition that the IO performance evaluation parameter P of the database is larger than a third preset evaluation parameter;
the second judging mode meets the condition that the IO performance evaluation parameter P of the database is larger than a second preset evaluation parameter and smaller than or equal to a third preset evaluation parameter;
the third judging mode meets the condition that the IO performance evaluation parameter P of the database is larger than the first preset evaluation parameter and smaller than or equal to the second preset evaluation parameter;
the fourth judging mode meets the condition that the IO performance evaluation parameter P of the database is smaller than or equal to a first preset evaluation parameter;
the third preset evaluation parameter is larger than the second preset evaluation parameter and larger than the first preset evaluation parameter.
Further, the central control module determines an adjustment mode for the early warning grade according to the proportion of the total interrupt duration of a plurality of transmission interrupt processes in the data transmission process of the acquired database to the preset duration under the second determination mode, wherein,
the first adjustment mode is that the central control module secondarily judges that the network environment of the server does not accord with a preset standard, and replaces the network working frequency band of the server corresponding to the database;
the second adjustment mode is that the central control module judges that the household appliance interferes with the network environment, adjusts the network risk early warning level to the first early warning level and sends out a first early warning notice aiming at the network fluctuation risk;
the third adjustment mode is that the central control module judges that the antenna transmitting power of the existing network supply end does not meet the preset requirement, adjusts the network risk early warning level to a second early warning level and sends out a second early warning notice aiming at the network fluctuation risk;
the first adjustment mode meets the condition that the proportion of the total interruption time length to the preset time length is smaller than or equal to the proportion of the first preset time length;
the second adjustment mode meets the condition that the proportion of the total interruption time length to the preset time length is smaller than or equal to the proportion of the second preset time length and larger than the proportion of the first preset time length;
The third adjustment mode meets the condition that the proportion of the total interruption time length to the preset time length is greater than that of the second preset time length;
wherein the first preset duration specific gravity is less than the second preset duration specific gravity.
Further, the central control module determines an adjustment mode of the total data amount in the database according to the difference value between the evaluation parameter P and the preset standard evaluation parameter in the third determination mode, wherein,
the first data volume adjusting mode is that the central control module adjusts the total data volume in the database to a preset total data volume, and stores overflow data into an auxiliary database of a newly built database;
the second data quantity adjusting mode is that the central control module adjusts the total data quantity in the database to a corresponding value by using a first adjusting coefficient, and stores overflow data into an auxiliary database of the newly built database;
the third data volume adjusting mode is that the central control module adjusts the total data volume in the database to a corresponding value by using a second adjusting coefficient, and stores overflow data into an auxiliary database of the newly built database;
the first data volume adjusting mode meets the condition that the difference value between the evaluation parameter P and the preset standard evaluation parameter is smaller than or equal to a first preset difference value;
The second data volume adjusting mode meets the condition that the difference value between the evaluation parameter P and the preset standard evaluation parameter is smaller than or equal to a second preset difference value and larger than the first preset difference value;
the third data volume adjustment mode meets the condition that the difference value between the evaluation parameter P and the preset standard evaluation parameter is larger than the second preset difference value;
the first preset difference value is smaller than the second preset difference value, and the first adjusting coefficient is smaller than the second adjusting coefficient.
Further, the central control module determines the regulation mode of the server according to the total amount of the newly added auxiliary databases in the acquired single server, wherein,
the first server adjusting mode is that the central control module judges that the running state of the server accords with a preset standard, and the central control module controls the server to run with the current running parameters;
the second server adjusting mode is that the central control module judges that the running state of the server does not accord with a preset standard, and the central control module adjusts the proportion of the reserved memory of the single database to the total memory of the server to a corresponding value according to the acquired evaluation parameter P of the single database and the average evaluation parameter values of a plurality of databases in the server;
the first server regulation mode meets the condition that the total amount of newly added auxiliary databases in the server is less than or equal to the preset standard amount;
The second server adjusting mode meets the condition that the total amount of the newly added auxiliary databases in the server is larger than the preset standard amount.
Further, the central control module calculates the specific gravity of the reserved memory of the single database accounting for the total memory of the server by using the specific gravity adjusting coefficient Q in a second server adjusting mode, wherein the calculation formula of Q is as follows:wherein,setting gamma=10 for average evaluation parameter values of a plurality of databases in a single server;
and the central control module uses the proportion adjustment coefficient to adjust the proportion of the reserved memory of each single database in the single server to the total memory of the server to a corresponding value.
Further, the central control module determines a secondary judgment mode aiming at the running state of the server according to the average evaluation parameter values of a plurality of databases in the single server under the second server regulation mode, wherein,
the first state secondary judgment mode is that the central control module judges that the running state of the server accords with a preset standard, and the central control module controls the server to run with the current running parameters;
the second state secondary judgment mode is that the central control module judges that the running state of the server does not accord with a preset standard, and determines an adjustment mode for server data according to the highest average evaluation parameter value and the lowest average evaluation parameter value in the acquired servers;
The third state secondary judgment mode is that the central control module judges that the network environment of the server does not accord with a preset standard;
the first state secondary judgment mode meets the condition that the average evaluation of a database in a server is larger than a second preset average evaluation;
the second state secondary judgment mode meets the condition that the average evaluation of a database in a server is larger than the first preset average evaluation and smaller than or equal to the second preset evaluation;
the third state secondary judgment mode meets the condition that the average evaluation of a database in a server is smaller than or equal to a first preset average evaluation;
wherein the first preset average rating is less than the second preset average rating.
Further, the central control module performs descending order arrangement on each server according to the average evaluation parameter values of a plurality of databases in a single server sequentially acquired in a second state secondary judgment mode to acquire the highest average evaluation parameter valueAnd the lowest mean evaluation parameter value->And determining an adjustment mode for the server data according to the evaluation difference value of the highest average evaluation parameter value and the lowest average evaluation parameter value, wherein,
the first server data adjustment mode is that the central control module coordinates the data of the server corresponding to the lowest average evaluation parameter value acquired by the central control module to a newly-built auxiliary server by using a first adjustment coefficient;
The second server data adjustment mode is that the central control module uses a second adjustment coefficient to adjust the data of the server corresponding to the highest average evaluation parameter value acquired by the central control module to the server corresponding to the lowest average evaluation parameter value;
the third server data adjustment mode is that the central control module uses a third adjustment coefficient to adjust the data of the server corresponding to the lowest average evaluation parameter value acquired by the central control module to the server corresponding to the highest average evaluation parameter value;
the first server data adjustment mode meets the condition that the difference value between the highest average evaluation parameter value and the lowest average evaluation parameter value is smaller than or equal to a first preset evaluation difference value;
the second server data adjustment mode meets the condition that the difference value between the highest average evaluation parameter value and the lowest average evaluation parameter value is smaller than or equal to a second preset evaluation difference value and larger than the first preset evaluation difference value;
the third server data adjustment mode meets the condition that the difference value between the highest average evaluation parameter value and the lowest average evaluation parameter value is larger than a second preset evaluation difference value;
the first preset evaluation difference value is smaller than the second preset evaluation difference value; the second adjustment factor is less than the third adjustment factor and less than the first adjustment factor.
Further, the central control module obtains the byte number read by the database for one time in the fourth judging mode, and determines an adjusting mode aiming at the preset evaluation parameter according to the difference value between the preset standard byte number and the byte number read by the database for one time, wherein,
the first adjusting mode is that the central control module respectively adjusts the first preset evaluation parameter, the second preset evaluation parameter and the third preset evaluation parameter to corresponding values by using a first preset evaluation parameter adjusting coefficient;
the second adjusting mode is that the central control module respectively adjusts the first preset evaluation parameter, the second preset evaluation parameter and the third preset evaluation parameter to corresponding values by using a second preset evaluation parameter adjusting coefficient;
the third adjusting mode is that the central control module adjusts the first preset evaluation parameter to a first corresponding evaluation parameter, adjusts the second preset evaluation parameter to a second corresponding evaluation parameter and adjusts the third preset evaluation parameter to a third corresponding evaluation parameter;
the first regulation mode meets the condition that the difference value between the number of preset standard bytes and the number of bytes read by the database in one time is smaller than or equal to a first preset byte difference value;
the second regulation mode satisfies that the difference value between the number of preset standard bytes and the number of bytes read by the database in one time is smaller than or equal to a second preset byte difference value and larger than the first preset byte difference value;
The third adjustment mode meets the condition that the difference between the number of preset standard bytes and the number of bytes read by the database once is larger than the second preset byte difference;
the first preset byte difference value is smaller than the second preset byte difference value, and the first preset evaluation adjustment coefficient is larger than the second preset evaluation adjustment coefficient.
Compared with the prior art, the method has the beneficial effects that the central control module calculates the database IO performance evaluation parameter P according to the number of single-time read bytes and the average transmission rate of a single database, judges whether the running state of the database accords with a preset standard according to the P, adjusts the alarm level to the corresponding alarm level according to the proportion of the total interrupt duration of a plurality of transmission interrupt processes of the database in the data transmission process to the preset duration, and adjusts the preset evaluation parameter of the database according to the byte number of the single-time read data of the database; the central control module adjusts overflow data of the database to an affiliated database of the database according to the IO performance evaluation parameter P of the database, and adjusts the proportion of reserved memory of the single database accounting for the total memory of the server according to the total amount of the newly added affiliated database and the average evaluation parameter values of a plurality of databases in the single server; after the comparison and readjustment, the central control module adjusts the data in the servers according to the average evaluation parameter value of the single server, the highest average evaluation parameter value and the lowest average evaluation parameter value in the plurality of servers, effectively improves the IO performance of the parallel data warehouse, further improves the running speed of the parallel distributed data warehouse, and performs planning processing on the electronic digital data so as to realize rationalization of resource allocation of the servers and the database.
Further, the central control module calculates the IO performance evaluation parameter P of the database according to the number of single read bytes of the single database and the average transmission rate, so that the running speed of the parallel distributed data warehouse is further improved while the IO performance of the single database during detection can be accurately obtained.
Further, the central control module determines whether the running state of the database accords with a judging mode of a preset standard according to P, and the running state of the database accords with the preset standard when P is larger than a third preset evaluation parameter; when the P is smaller than or equal to the third preset evaluation parameter and larger than the second preset evaluation parameter, the central control module judges that the network environment of the database does not accord with a preset standard; if P is smaller than or equal to the second preset evaluation parameter and larger than the first preset evaluation parameter, the central control module judges that a single database is overloaded; if the evaluation parameter P is smaller than or equal to the first preset evaluation parameter, the central control module judges that the IO basic performance of the database does not accord with a preset standard; the method and the device further improve the running speed of the parallel distributed data warehouse while ensuring that the reasons of poor running state of the database are accurately acquired.
Further, when the network environment of the server is judged not to meet the preset standard, the central control module determines an adjusting mode for the alarm level according to the proportion of the total interrupt duration of a plurality of transmission interrupt processes in the data transmission process of the acquired database to the preset duration, and the running speed of the parallel distributed data warehouse is further improved while ensuring that the server works in a stable network state and timely discovers the reason of fluctuation of the network.
Further, when the central control module judges that P is smaller than or equal to a second preset evaluation parameter and larger than a first preset evaluation parameter, the total data amount of the database is adjusted to a corresponding value according to the difference value between P and a preset standard evaluation parameter, the data are arranged in a descending order according to the storage time, and premature storage data exceeding the adjusted total data amount are transferred and stored into a newly-built auxiliary database corresponding to the database; the running speed of the parallel distributed data warehouse is further improved while the IO performance of the database is not reduced due to excessive stored data.
Furthermore, the central control module determines the regulation mode of the server according to the total amount of the newly added auxiliary databases in the single server, thereby effectively avoiding the reduction of the data transmission rate caused by excessive newly added databases and excessive server load and further improving the running speed of the parallel distributed data warehouse.
Further, the central control module calculates the proportion of the reserved memory of the single database accounting for the total memory of the server by using the proportion adjusting coefficient Q in the second server adjusting mode, so that the running speed of the parallel distributed data warehouse is further improved while the reasonable memory distribution is ensured.
Further, the central control module adjusts the proportion of the reserved memory of each single database in the single server to the total memory of the server to a corresponding value, and then determines a secondary judgment mode for the running state of the server according to the obtained average evaluation parameter values of a plurality of databases in the single server, so that the running speed of the parallel distributed data warehouse is further improved while the reasonable planning of the data is ensured.
Further, the central control module is used for obtaining the highest average evaluation parameter value according to the second server secondary judgment modeAnd the lowest mean evaluation parameter value->The difference value of the data is determined for the adjustment mode of the server data, and the IO performance of the parallel data warehouse is further improved while the memory allocation is further rationalized.
Further, when the central control module judges that the IO basic performance of the server corresponding to the database is too low, the preset evaluation parameters are reduced to the corresponding values according to the difference value between the preset standard byte number and the byte number read by the database once, and the operation speed of the parallel distributed data warehouse is further improved while the application range of the method is wider.
Drawings
FIG. 1 is a flowchart illustrating steps of a method for enhancing IO performance of a parallel data warehouse using distributed virtual storage in accordance with an embodiment of the present invention;
FIG. 2 is a flowchart of a method for determining a database running state by using a method for improving IO performance of a parallel data warehouse by using distributed virtual storage according to an embodiment of the present invention;
FIG. 3 is a flow chart of a method for improving IO performance of a parallel data warehouse using distributed virtual storage in a manner of adjusting total data in a database according to an embodiment of the present invention;
FIG. 4 is a flow chart of a state secondary decision method for improving IO performance of a parallel data warehouse using distributed virtual storage in accordance with an embodiment of the present invention.
Detailed Description
In order that the objects and advantages of the invention will become more apparent, the invention will be further described with reference to the following examples; it should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are merely for explaining the technical principles of the present invention, and are not intended to limit the scope of the present invention.
It should be noted that, in the description of the present invention, terms such as "upper," "lower," "left," "right," "inner," "outer," and the like indicate directions or positional relationships based on the directions or positional relationships shown in the drawings, which are merely for convenience of description, and do not indicate or imply that the apparatus or elements must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the present invention.
Furthermore, it should be noted that, in the description of the present invention, unless explicitly specified and limited otherwise, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be either fixedly connected, detachably connected, or integrally connected, for example; can be mechanically or electrically connected; can be directly connected or indirectly connected through an intermediate medium, and can be communication between two elements. The specific meaning of the above terms in the present invention can be understood by those skilled in the art according to the specific circumstances.
Referring to fig. 1, fig. 2, fig. 3, and fig. 4, which are respectively a step flowchart of a method for improving IO performance of a parallel data warehouse by using distributed virtual storage, a decision mode flowchart of a database running state, an adjustment mode flowchart of total data amount in a database, and a state secondary decision mode flowchart according to an embodiment of the present invention; the embodiment of the invention discloses a method for improving IO performance of a parallel data warehouse by using distributed virtual storage, which comprises the following steps:
step S1, a central control module calculates a database IO performance evaluation parameter P according to the byte number and the average transmission rate read by a single database, judges whether the running state of the database meets a preset standard according to the P, and a parallel distributed data warehouse comprises a plurality of servers, wherein the single server comprises a plurality of databases for storing data in a preset storage path;
Step S2, the central control module adjusts the early warning level to the corresponding alarm level according to the proportion of the calculated total interrupt duration of a plurality of transmission interrupt processes occurring in the data transmission process of the database in the preset duration, and adjusts the preset evaluation parameters of the database according to the byte number of the data read by the database once;
step S3, the central control module adjusts overflow data of the database to an auxiliary database of the database according to the IO performance evaluation parameter P of the database, and adjusts the proportion of reserved memory of the single database to the total memory of the server according to the total amount of the newly added auxiliary database and the average evaluation parameter values of a plurality of databases in the single server;
and S4, after the comparison and the readjustment, the central control module adjusts the data in the servers according to the acquired average evaluation parameter value of the single server, the highest average evaluation parameter value and the lowest average evaluation parameter value in the plurality of servers.
Specifically, in the step S1, the central control module calculates the database IO performance evaluation parameters P, P according to the number of bytes read by a single database and the average transmission rate, where the calculation formula of P is, Wherein D is i For the average transmission rate of the database in the ith preset period, i=1, 2,3 … n is set, Z is the number of bytes read by a single database at a time, alpha is a first evaluation correction coefficient, alpha=0.05 s is set, beta is a second evaluation correction coefficient, and beta=50 kb is set -1 。
Specifically, the central control module determines whether the running state of the database accords with the judging mode of the preset standard according to the IO performance evaluation parameter P of the database, wherein,
the first judging mode is that the central control module judges that the running state of the database meets the preset standard and controls the database to run with the current running parameters;
the second judging mode is that the central control module judges that the network environment of the server corresponding to the database does not accord with a preset standard, invokes the total interrupt duration of a plurality of transmission interrupt processes of the database in the data transmission process to calculate the proportion of the total interrupt duration in the preset duration, and judges whether the network environment of the server accords with the preset standard or not according to the calculation result;
the third judging mode is that the central control module judges that the running state of the database does not accord with a preset standard, and the central control module obtains the total data quantity in the database and adjusts the total data quantity of the database to a corresponding value according to the difference value between the evaluation parameter P and the preset standard evaluation parameter;
The fourth judging mode is that the central control module judges that the IO basic performance of the database does not accord with a preset standard, and the central control module sends out alarm information aiming at replacing corresponding hardware and reduces the preset evaluation parameter to a corresponding value;
the first judging mode meets the condition that the IO performance evaluation parameter P of the database is larger than a third preset evaluation parameter;
the second judging mode meets the condition that the IO performance evaluation parameter P of the database is larger than a second preset evaluation parameter and smaller than or equal to a third preset evaluation parameter;
the third judging mode meets the condition that the IO performance evaluation parameter P of the database is larger than the first preset evaluation parameter and smaller than or equal to the second preset evaluation parameter;
the fourth judging mode meets the condition that the IO performance evaluation parameter P of the database is smaller than or equal to a first preset evaluation parameter;
the third preset evaluation parameter is larger than the second preset evaluation parameter and larger than the first preset evaluation parameter.
Specifically, the central control module determines an adjustment mode for the early warning level according to the proportion of the total interrupt duration of a plurality of transmission interrupt processes in the data transmission process of the acquired database to the preset duration in a second determination mode, wherein,
the first adjustment mode is that the central control module secondarily judges that the network environment of the server does not accord with a preset standard, and replaces the network working frequency band of the server corresponding to the database;
The second adjustment mode is that the central control module judges that the household appliance interferes with the network environment, adjusts the network risk early warning level to the first early warning level and sends out a first early warning notice aiming at the network fluctuation risk;
the third adjustment mode is that the central control module judges that the antenna transmitting power of the existing network supply end does not meet the preset requirement, adjusts the network risk early warning level to a second early warning level and sends out a second early warning notice aiming at the network fluctuation risk;
the first adjustment mode meets the condition that the proportion of the total interruption time length to the preset time length is smaller than or equal to the proportion of the first preset time length;
the second adjustment mode meets the condition that the proportion of the total interruption time length to the preset time length is smaller than or equal to the proportion of the second preset time length and larger than the proportion of the first preset time length;
the third adjustment mode meets the condition that the proportion of the total interruption time length to the preset time length is greater than that of the second preset time length;
wherein the first preset duration specific gravity is less than the second preset duration specific gravity.
Specifically, the central control module determines an adjustment mode of the total data amount in the database according to the difference value between the evaluation parameter P and the preset standard evaluation parameter in the third determination mode, wherein,
the first data volume adjusting mode is that the central control module adjusts the total data volume in the database to a preset total data volume, and stores overflow data into an auxiliary database of a newly built database;
The second data quantity adjusting mode is that the central control module adjusts the total data quantity in the database to a corresponding value by using a first adjusting coefficient, and stores overflow data into an auxiliary database of the newly built database;
the third data volume adjusting mode is that the central control module adjusts the total data volume in the database to a corresponding value by using a second adjusting coefficient, and stores overflow data into an auxiliary database of the newly built database;
the first data volume adjusting mode meets the condition that the difference value between the evaluation parameter P and the preset standard evaluation parameter is smaller than or equal to a first preset difference value;
the second data volume adjusting mode meets the condition that the difference value between the evaluation parameter P and the preset standard evaluation parameter is smaller than or equal to a second preset difference value and larger than the first preset difference value;
the third data volume adjustment mode meets the condition that the difference value between the evaluation parameter P and the preset standard evaluation parameter is larger than the second preset difference value;
the first preset difference value is smaller than the second preset difference value, and the first adjusting coefficient is smaller than the second adjusting coefficient.
Specifically, the central control module determines the regulation mode of the server according to the total amount of the newly added auxiliary databases in the acquired single server, wherein,
the first server adjusting mode is that the central control module judges that the running state of the server accords with a preset standard, and the central control module controls the server to run with the current running parameters;
The second server adjusting mode is that the central control module judges that the running state of the server does not accord with a preset standard, and the central control module adjusts the proportion of the reserved memory of the single database to the total memory of the server to a corresponding value according to the acquired evaluation parameter P of the single database and the average evaluation parameter values of a plurality of databases in the server;
the first server regulation mode meets the condition that the total amount of newly added auxiliary databases in the server is less than or equal to the preset standard amount;
the second server adjusting mode meets the condition that the total amount of the newly added auxiliary databases in the server is larger than the preset standard amount.
Specifically, the central control module calculates the specific gravity of the reserved memory of the single database accounting for the total memory of the server by using a specific gravity adjusting coefficient Q in a second server adjusting mode, wherein the calculation formula of Q is as follows:wherein,setting gamma=10 for average evaluation parameter values of a plurality of databases in a single server;
and the central control module uses the proportion adjustment coefficient to adjust the proportion of the reserved memory of each single database in the single server to the total memory of the server to a corresponding value.
Specifically, the central control module determines a secondary judgment mode aiming at the running state of the server according to the average evaluation parameter values of a plurality of databases in a single server under the second server regulation mode, wherein,
The first state secondary judgment mode is that the central control module judges that the running state of the server accords with a preset standard, and the central control module controls the server to run with the current running parameters;
the second state secondary judgment mode is that the central control module judges that the running state of the server does not accord with a preset standard, and determines an adjustment mode for server data according to the highest average evaluation parameter value and the lowest average evaluation parameter value in the acquired servers;
the third state secondary judgment mode is that the central control module judges that the network environment of the server does not accord with a preset standard;
the first state secondary judgment mode meets the condition that the average evaluation of a database in a server is larger than a second preset average evaluation;
the second state secondary judgment mode meets the condition that the average evaluation of a database in a server is larger than the first preset average evaluation and smaller than or equal to the second preset evaluation;
the third state secondary judgment mode meets the condition that the average evaluation of a database in a server is smaller than or equal to a first preset average evaluation;
wherein the first preset average rating is less than the second preset average rating.
Specifically, the central control module performs descending order arrangement on each server according to average evaluation parameter values of a plurality of databases in a single server sequentially acquired in a second state secondary judgment mode to acquire the highest average evaluation parameter value And the lowest mean evaluation parameter value->And determining an adjustment mode for the server data according to the evaluation difference value of the highest average evaluation parameter value and the lowest average evaluation parameter value, wherein,
the first server data adjustment mode is that the central control module coordinates the data of the server corresponding to the lowest average evaluation parameter value acquired by the central control module to a newly-built auxiliary server by using a first adjustment coefficient;
the second server data adjustment mode is that the central control module uses a second adjustment coefficient to adjust the data of the server corresponding to the highest average evaluation parameter value acquired by the central control module to the server corresponding to the lowest average evaluation parameter value;
the third server data adjustment mode is that the central control module uses a third adjustment coefficient to adjust the data of the server corresponding to the lowest average evaluation parameter value acquired by the central control module to the server corresponding to the highest average evaluation parameter value;
the first server data adjustment mode meets the condition that the difference value between the highest average evaluation parameter value and the lowest average evaluation parameter value is smaller than or equal to a first preset evaluation difference value;
the second server data adjustment mode meets the condition that the difference value between the highest average evaluation parameter value and the lowest average evaluation parameter value is smaller than or equal to a second preset evaluation difference value and larger than the first preset evaluation difference value;
The third server data adjustment mode meets the condition that the difference value between the highest average evaluation parameter value and the lowest average evaluation parameter value is larger than a second preset evaluation difference value;
the first preset evaluation difference value is smaller than the second preset evaluation difference value, the second adjusting coefficient is smaller than the third adjusting coefficient and is smaller than the first adjusting coefficient.
Specifically, the central control module obtains the byte number read by the database for one time in the fourth judging mode, and determines an adjusting mode aiming at a preset evaluation parameter according to the difference value between the preset standard byte number and the byte number read by the database for one time, wherein,
the first adjusting mode is that the central control module respectively adjusts the first preset evaluation parameter, the second preset evaluation parameter and the third preset evaluation parameter to corresponding values by using a first preset evaluation parameter adjusting coefficient;
the second adjusting mode is that the central control module respectively adjusts the first preset evaluation parameter, the second preset evaluation parameter and the third preset evaluation parameter to corresponding values by using a second preset evaluation parameter adjusting coefficient;
the third adjusting mode is that the central control module adjusts the first preset evaluation parameter to a first corresponding evaluation parameter, adjusts the second preset evaluation parameter to a second corresponding evaluation parameter and adjusts the third preset evaluation parameter to a third corresponding evaluation parameter;
The first regulation mode meets the condition that the difference value between the number of preset standard bytes and the number of bytes read by the database in one time is smaller than or equal to a first preset byte difference value;
the second regulation mode satisfies that the difference value between the number of preset standard bytes and the number of bytes read by the database in one time is smaller than or equal to a second preset byte difference value and larger than the first preset byte difference value;
the third adjustment mode meets the condition that the difference between the number of preset standard bytes and the number of bytes read by the database once is larger than the second preset byte difference;
the first preset byte difference value is smaller than the second preset byte difference value, and the first preset evaluation adjustment coefficient is larger than the second preset evaluation adjustment coefficient. Thus far, the technical solution of the present invention has been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of protection of the present invention is not limited to these specific embodiments. Equivalent modifications and substitutions for related technical features may be made by those skilled in the art without departing from the principles of the present invention, and such modifications and substitutions will be within the scope of the present invention.
The foregoing description is only of the preferred embodiments of the invention and is not intended to limit the invention; various modifications and variations of the present invention will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method for improving IO performance of a parallel data warehouse using distributed virtual storage, comprising:
step S1, a central control module calculates a database IO performance evaluation parameter P according to the number of bytes read by a single database and the average transmission rate, and judges whether the running state of the database meets a preset standard according to the P;
step S2, the central control module adjusts the early warning level to the corresponding alarm level according to the proportion of the calculated total interrupt duration of a plurality of transmission interrupt processes occurring in the data transmission process of the database in the preset duration, and adjusts the preset evaluation parameters of the database according to the byte number of the data read by the database once;
step S3, the central control module adjusts overflow data of the database to an auxiliary database of the database according to the IO performance evaluation parameter P of the database, and adjusts the proportion of reserved memory of the single database to the total memory of the server according to the total amount of the newly added auxiliary database and the average evaluation parameter values of a plurality of databases in the single server;
and S4, after the comparison and the readjustment, the central control module adjusts the data in the servers according to the acquired average evaluation parameter value of the single server, the highest average evaluation parameter value and the lowest average evaluation parameter value in the plurality of servers.
2. The method for improving IO performance of a parallel data warehouse using distributed virtual storage as claimed in claim 1, wherein in step S1, the central control module calculates database IO performance evaluation parameters P, P according to the number of bytes read by a single database and the average transmission rate, wherein the calculation formula of P is,wherein D is i For the average transmission rate of the database in the ith preset period, i=1, 2,3 … n is set, Z is the number of bytes read by a single database at a time, alpha is a first evaluation correction coefficient, alpha=0.05 s is set, beta is a second evaluation correction coefficient, and beta=50 kb is set -1 。
3. The method for improving the IO performance of the parallel data warehouse by using the distributed virtual storage as claimed in claim 2, wherein the central control module determines whether the running state of the database meets the judgment mode of the preset standard according to the IO performance evaluation parameter P of the database, wherein,
the first judging mode is that the central control module judges that the running state of the database meets the preset standard and controls the database to run with the current running parameters;
the second judging mode is that the central control module judges that the network environment of the server corresponding to the database does not accord with a preset standard, invokes the total interrupt duration of a plurality of transmission interrupt processes of the database in the data transmission process to calculate the proportion of the total interrupt duration in the preset duration, and judges whether the network environment of the server accords with the preset standard or not according to the calculation result;
The third judging mode is that the central control module judges that the running state of the database does not accord with a preset standard, and the central control module obtains the total data quantity in the database and adjusts the total data quantity of the database to a corresponding value according to the difference value between the evaluation parameter P and the preset standard evaluation parameter;
the fourth judging mode is that the central control module judges that the IO basic performance of the database does not accord with a preset standard, and the central control module sends out alarm information aiming at replacing corresponding hardware and reduces the preset evaluation parameter to a corresponding value;
the first judging mode meets the condition that the IO performance evaluation parameter P of the database is larger than a third preset evaluation parameter;
the second judging mode meets the condition that the IO performance evaluation parameter P of the database is larger than a second preset evaluation parameter and smaller than or equal to a third preset evaluation parameter;
the third judging mode meets the condition that the IO performance evaluation parameter P of the database is larger than the first preset evaluation parameter and smaller than or equal to the second preset evaluation parameter;
the fourth judging mode meets the condition that the IO performance evaluation parameter P of the database is smaller than or equal to a first preset evaluation parameter;
the third preset evaluation parameter is larger than the second preset evaluation parameter and larger than the first preset evaluation parameter.
4. The method for improving IO performance of a parallel data warehouse by using distributed virtual storage according to claim 3, wherein the central control module determines an adjustment mode for the early warning level according to the proportion of the total interrupt duration of a plurality of transmission interrupt processes in the data transmission process of the acquired database to the preset duration in a second determination mode, wherein,
the first adjustment mode is that the central control module secondarily judges that the network environment of the server does not accord with a preset standard, and replaces the network working frequency band of the server corresponding to the database;
the second adjustment mode is that the central control module judges that the household appliance interferes with the network environment, adjusts the network risk early warning level to the first early warning level and sends out a first early warning notice aiming at the network fluctuation risk;
the third adjustment mode is that the central control module judges that the antenna transmitting power of the existing network supply end does not meet the preset requirement, adjusts the network risk early warning level to a second early warning level and sends out a second early warning notice aiming at the network fluctuation risk;
the first adjustment mode meets the condition that the proportion of the total interruption time length to the preset time length is smaller than or equal to the proportion of the first preset time length;
the second adjustment mode meets the condition that the proportion of the total interruption time length to the preset time length is smaller than or equal to the proportion of the second preset time length and larger than the proportion of the first preset time length;
The third adjustment mode meets the condition that the proportion of the total interruption time length to the preset time length is greater than that of the second preset time length;
wherein the first preset duration specific gravity is less than the second preset duration specific gravity.
5. The method for improving IO performance of a parallel data warehouse using distributed virtual storage as claimed in claim 4, wherein the central control module determines an adjustment mode of total data amount in the database according to a difference between the evaluation parameter P and a preset standard evaluation parameter in the third determination mode, wherein,
the first data volume adjusting mode is that the central control module adjusts the total data volume in the database to a preset total data volume, and stores overflow data into an auxiliary database of a newly built database;
the second data quantity adjusting mode is that the central control module adjusts the total data quantity in the database to a corresponding value by using a first adjusting coefficient, and stores overflow data into an auxiliary database of the newly built database;
the third data volume adjusting mode is that the central control module adjusts the total data volume in the database to a corresponding value by using a second adjusting coefficient, and stores overflow data into an auxiliary database of the newly built database;
the first data volume adjusting mode meets the condition that the difference value between the evaluation parameter P and the preset standard evaluation parameter is smaller than or equal to a first preset difference value;
The second data volume adjusting mode meets the condition that the difference value between the evaluation parameter P and the preset standard evaluation parameter is smaller than or equal to a second preset difference value and larger than the first preset difference value;
the third data volume adjustment mode meets the condition that the difference value between the evaluation parameter P and the preset standard evaluation parameter is larger than the second preset difference value;
the first preset difference value is smaller than the second preset difference value, and the first adjusting coefficient is smaller than the second adjusting coefficient.
6. The method for improving IO performance of a parallel data warehouse using distributed virtual storage as claimed in claim 5, wherein the central control module determines the adjustment mode of the server according to the total amount of newly added auxiliary databases in the acquired single server, wherein,
the first server adjusting mode is that the central control module judges that the running state of the server accords with a preset standard, and the central control module controls the server to run with the current running parameters;
the second server adjusting mode is that the central control module judges that the running state of the server does not accord with a preset standard, and the central control module adjusts the proportion of the reserved memory of the single database to the total memory of the server to a corresponding value according to the acquired evaluation parameter P of the single database and the average evaluation parameter values of a plurality of databases in the server;
The first server regulation mode meets the condition that the total amount of newly added auxiliary databases in the server is less than or equal to the preset standard amount;
the second server adjusting mode meets the condition that the total amount of the newly added auxiliary databases in the server is larger than the preset standard amount.
7. The method of claim 6The method for improving IO performance of the parallel data warehouse by using the distributed virtual storage is characterized in that the central control module calculates the proportion of the reserved memory of the single database to the total memory of the server by using a proportion adjustment coefficient Q in a second server adjustment mode, wherein the calculation formula of Q is as follows:wherein (1)>Setting gamma=10 for average evaluation parameter values of a plurality of databases in a single server;
and the central control module uses the proportion adjustment coefficient to adjust the proportion of the reserved memory of each single database in the single server to the total memory of the server to a corresponding value.
8. The method for improving IO performance of a parallel data warehouse using distributed virtual storage as claimed in claim 7, wherein the central control module determines a secondary decision mode for the server operational status based on the re-acquired average evaluation parameter values of the plurality of databases within the single server in the second server tuning mode, wherein,
The first state secondary judgment mode is that the central control module judges that the running state of the server accords with a preset standard, and the central control module controls the server to run with the current running parameters;
the second state secondary judgment mode is that the central control module judges that the running state of the server does not accord with a preset standard, and determines an adjustment mode for server data according to the highest average evaluation parameter value and the lowest average evaluation parameter value in the acquired servers;
the third state secondary judgment mode is that the central control module judges that the network environment of the server does not accord with a preset standard;
the first state secondary judgment mode meets the condition that the average evaluation of a database in a server is larger than a second preset average evaluation;
the second state secondary judgment mode meets the condition that the average evaluation of a database in a server is larger than the first preset average evaluation and smaller than or equal to the second preset evaluation;
the third state secondary judgment mode meets the condition that the average evaluation of a database in a server is smaller than or equal to a first preset average evaluation;
wherein the first preset average rating is less than the second preset average rating.
9. The method for improving IO performance of a parallel data warehouse using distributed virtual storage as claimed in claim 8, wherein the central control module performs descending order of servers according to average evaluation parameter values of a plurality of databases in a single server sequentially acquired in a second state secondary judgment mode to acquire a highest average evaluation parameter value And the lowest mean evaluation parameter value->And determining an adjustment mode for the server data according to the difference value between the highest average evaluation parameter value and the lowest average evaluation parameter value, wherein,
the first server data adjustment mode is that the central control module coordinates the data of the server corresponding to the lowest average evaluation parameter value acquired by the central control module to a newly-built auxiliary server by using a first adjustment coefficient;
the second server data adjustment mode is that the central control module uses a second adjustment coefficient to adjust the data of the server corresponding to the highest average evaluation parameter value acquired by the central control module to the server corresponding to the lowest average evaluation parameter value;
the third server data adjustment mode is that the central control module uses a third adjustment coefficient to adjust the data of the server corresponding to the lowest average evaluation parameter value acquired by the central control module to the server corresponding to the highest average evaluation parameter value;
the first server data adjustment mode meets the condition that the difference value between the highest average evaluation parameter value and the lowest average evaluation parameter value is smaller than or equal to a first preset evaluation difference value;
the second server data adjustment mode meets the condition that the difference value between the highest average evaluation parameter value and the lowest average evaluation parameter value is smaller than or equal to a second preset evaluation difference value and larger than the first preset evaluation difference value;
The third server data adjustment mode meets the condition that the difference value between the highest average evaluation parameter value and the lowest average evaluation parameter value is larger than a second preset evaluation difference value;
the first preset evaluation difference value is smaller than the second preset evaluation difference value, the second adjusting coefficient is smaller than the third adjusting coefficient and is smaller than the first adjusting coefficient.
10. The method for improving IO performance of a parallel data warehouse using distributed virtual storage as claimed in claim 9, wherein the central control module obtains the number of bytes read by the database once in the fourth determination mode, and determines an adjustment mode for a preset evaluation parameter according to a difference between the preset standard number of bytes and the number of bytes read by the database once,
the first adjusting mode is that the central control module respectively adjusts the first preset evaluation parameter, the second preset evaluation parameter and the third preset evaluation parameter to corresponding values by using a first preset evaluation parameter adjusting coefficient;
the second adjusting mode is that the central control module respectively adjusts the first preset evaluation parameter, the second preset evaluation parameter and the third preset evaluation parameter to corresponding values by using a second preset evaluation parameter adjusting coefficient;
the third adjusting mode is that the central control module adjusts the first preset evaluation parameter to a first corresponding evaluation parameter, adjusts the second preset evaluation parameter to a second corresponding evaluation parameter and adjusts the third preset evaluation parameter to a third corresponding evaluation parameter;
The first regulation mode meets the condition that the difference value between the number of preset standard bytes and the number of bytes read by the database in one time is smaller than or equal to a first preset byte difference value;
the second regulation mode satisfies that the difference value between the number of preset standard bytes and the number of bytes read by the database in one time is smaller than or equal to a second preset byte difference value and larger than the first preset byte difference value;
the third adjustment mode meets the condition that the difference between the number of preset standard bytes and the number of bytes read by the database once is larger than the second preset byte difference;
the first preset byte difference value is smaller than the second preset byte difference value, and the first preset evaluation adjustment coefficient is larger than the second preset evaluation adjustment coefficient.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310162137.6A CN117235189A (en) | 2023-02-15 | 2023-02-15 | Method for improving IO performance of parallel data warehouse by using distributed virtual storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310162137.6A CN117235189A (en) | 2023-02-15 | 2023-02-15 | Method for improving IO performance of parallel data warehouse by using distributed virtual storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117235189A true CN117235189A (en) | 2023-12-15 |
Family
ID=89091779
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310162137.6A Pending CN117235189A (en) | 2023-02-15 | 2023-02-15 | Method for improving IO performance of parallel data warehouse by using distributed virtual storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117235189A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117692530A (en) * | 2024-02-02 | 2024-03-12 | 中铁四局集团有限公司 | Multi-protocol access system and method for multi-Internet of things equipment |
-
2023
- 2023-02-15 CN CN202310162137.6A patent/CN117235189A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117692530A (en) * | 2024-02-02 | 2024-03-12 | 中铁四局集团有限公司 | Multi-protocol access system and method for multi-Internet of things equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110967645B (en) | SOC correction method and apparatus, battery management system, and storage medium | |
| US8112555B2 (en) | Peripheral adapter interrupt frequency control by estimating processor load at the peripheral adapter | |
| CN117235189A (en) | Method for improving IO performance of parallel data warehouse by using distributed virtual storage | |
| JP2017112655A (en) | Power storage system management device, power storage system management method, power storage system | |
| MXPA04001474A (en) | Method and apparatus for scheduling packet data transmissions in a wireless communication system. | |
| CN110636109B (en) | Node scheduling optimization method, server and computer readable storage medium | |
| CN115222237A (en) | Power system inertia demand assessment method based on space-time characteristics and multi-index constraints | |
| CN107218702A (en) | Air conditioner and air conditioner frequency adjustment method and computer-readable recording medium | |
| CN114785565A (en) | Data security exchange system based on network boundary | |
| CN117148391B (en) | Beidou satellite navigation credible position service system and method | |
| CN112834818A (en) | Electric quantity determining method and device, storage medium and electronic equipment | |
| CN102845028B (en) | Method and device for resource allocation | |
| CN115540273A (en) | Compressor frequency adjusting method and device, air conditioning equipment and storage medium | |
| CN110895451A (en) | Data access performance optimization method based on distributed system | |
| CN114205247B (en) | Access method and device of power distribution Internet of things, computer equipment and storage medium | |
| CN108092910B (en) | Data traffic scheduling method and device | |
| CN110572356A (en) | Computing power migration method and system based on edge gateway data quality evaluation | |
| CN115829786B (en) | Power grid response adjusting method based on air conditioner load | |
| CN110650217B (en) | Load balancing method for download server | |
| CN115949560B (en) | Method, device, equipment and medium for adaptively controlling cabin temperature of wind turbine generator | |
| CN115604276B (en) | Intelligent management system based on server resource utilization | |
| CN116980417A (en) | CDN node scheduling method and device | |
| CN115758373A (en) | Method for unified nanotube management of multiple cloud servers for cloud management | |
| CN116624967A (en) | Compressor frequency control method, device, equipment and storage medium | |
| CN115174678A (en) | Edge caching method for intelligent monitoring platform of electrical equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |