CN111444184A - Data storage method, storage scheduling server, electronic device and storage medium - Google Patents
Data storage method, storage scheduling server, electronic device and storage medium Download PDFInfo
- Publication number
- CN111444184A CN111444184A CN202010220039.XA CN202010220039A CN111444184A CN 111444184 A CN111444184 A CN 111444184A CN 202010220039 A CN202010220039 A CN 202010220039A CN 111444184 A CN111444184 A CN 111444184A
- Authority
- CN
- China
- Prior art keywords
- database
- data
- query performance
- performance
- storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/256—Integrating or interfacing systems involving database management systems in federated or virtual databases
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data storage method, a storage scheduling server, electronic equipment and a storage medium, wherein the method comprises the following steps: receiving data; respectively monitoring the query performance of the first database and the query performance of the second database; if the query performance of the first database is higher than that of the second database, or the query performance of the first database is the same as that of the second database, storing the data into the first database and the second database; and if the query performance of the second database is higher than that of the first database, stopping storing the data in the first database. The database is selected according to the query performance, the technical problem that the database cannot adapt to the data storage requirement in the later period, so that the query performance of the database is poor can be solved, and the technical effect of improving the query performance of the database can be achieved.
Description
Technical Field
The present invention relates to the field of database technologies, and in particular, to a data storage method, a storage scheduling server, an electronic device, and a storage medium.
Background
With the generation of various targeted application services, both log data generated by applications and business data accessed by applications require more powerful databases for storage.
In the prior art, a proper database can be selected for data storage according to business characteristics, but as time goes on, the database selected at the beginning may not be suitable for the later data storage requirement, so that the query performance of the database is poor.
Disclosure of Invention
The embodiment of the invention aims to provide a data storage method, a storage scheduling server, electronic equipment and a storage medium, so as to realize better query performance. The specific technical scheme is as follows:
in a first aspect of the present invention, there is provided a data storage method applied to a storage scheduling server, where the method includes:
receiving data, and respectively monitoring the query performance of a first database and the query performance of a second database;
if the query performance of the first database is higher than that of the second database, or the query performance of the first database is the same as that of the second database, storing the data into the first database and the second database; the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the second database is higher than that of the first database after the accumulated amount of the received data reaches the preset data amount threshold;
and if the query performance of the second database is higher than that of the first database, stopping storing the data in the first database.
Optionally, before the monitoring the query performance of the first database and the query performance of the second database, respectively, the method further includes:
if the target database exists, storing the data into the target database;
if the target database does not exist, the step of respectively monitoring the query performance of the first database and the query performance of the second database is carried out;
after the ceasing to store the data to the first database, the method further comprises:
and taking the second database as a target database.
Optionally, after the receiving data, the method further comprises:
determining an accumulated amount of received data;
if the query performance of the second database is higher than the query performance of the first database, stopping storing the data in the first database, including:
and if the query performance of the second database is higher than that of the first database and the accumulated amount is greater than or equal to a preset data amount threshold value, stopping storing the data in the first database.
Optionally, after stopping storing the data in the first database, the method further includes:
and recovering the storage resources of the first database.
Optionally, the first database is an RDBMS database, and the second database is an O L AP database.
In a second aspect of the present invention, there is also provided a storage scheduling server, including:
the data receiving module is used for receiving data;
the performance monitoring module is used for respectively monitoring the query performance of the first data and the query performance of the second data;
the first storage module is used for storing the data into the first database and the second database if the query performance of the first database is higher than that of the second database or the query performance of the first database is the same as that of the second database; the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the second database is higher than that of the first database after the accumulated amount of the received data reaches the preset data amount threshold;
and the second storage module is used for stopping storing the data to the first database if the query performance of the second database is higher than that of the first database.
Optionally, the server further comprises:
the third storage module is used for storing the data into a target database if the target database exists;
the performance monitoring entering module is used for entering the performance monitoring module if the target database does not exist;
and the target database determining module is used for taking the second database as a target database.
Optionally, the server further comprises:
the accumulated amount determining module is used for determining the accumulated amount of the received data;
the second storage module is further configured to:
and if the query performance of the second database is higher than that of the first database and the accumulated amount is greater than or equal to a preset data amount threshold value, stopping storing the data in the first database.
Optionally, the server further comprises:
and the resource recovery module is used for recovering the storage resources of the first database.
Optionally, the first database is an RDBMS database, and the second database is an O L AP database.
In yet another aspect of the present invention, there is also provided a computer-readable storage medium having stored therein instructions, which when run on a computer, cause the computer to perform any of the data storage methods described above.
In yet another aspect of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform any of the data storage methods described above.
The data storage method, the storage scheduling server, the electronic device and the storage medium provided by the embodiment of the invention receive data; respectively monitoring the query performance of the first database and the query performance of the second database; if the query performance of the first database is higher than that of the second database, or the query performance of the first database is the same as that of the second database, storing the data into the first database and the second database; the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the second database is higher than that of the first database after the accumulated amount of the received data reaches the preset data amount threshold; and if the query performance of the second database is higher than that of the first database, stopping storing the data in the first database. The database is selected according to the query performance, the technical problem that the database cannot adapt to the data storage requirement in the later period, so that the query performance of the database is poor can be solved, and the technical effect of improving the query performance of the database can be achieved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
FIG. 1 is a flow chart illustrating steps of a data storage method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating interaction among a storage scheduling server, a database, and an application layer according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a relationship between a response speed and a cumulative amount of received data according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating a relationship between a response time duration and a cumulative amount of received data according to an embodiment of the present invention;
FIG. 5 is a flow chart illustrating steps of another data storage method according to an embodiment of the present invention;
FIG. 6 is a block diagram of a storage scheduling server according to an embodiment of the present invention;
FIG. 7 is a block diagram of another storage scheduling server according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device in an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.
The method of the embodiment of the invention can be applied to a storage scheduling server, the storage scheduling server is used for receiving the data and storing the data into the first database and the second database, and the storage scheduling server can also perform preprocessing such as extraction, cleaning, conversion, loading and the like on the data before storing the data, so that the storage scheduling server can also be called an ET L (Extract Transform L oad) server.
Example one
Referring to fig. 1, a flowchart illustrating steps of a data storage method according to an embodiment of the present invention is shown, which specifically includes the following steps:
The embodiment of the present invention may be used for a storage scheduling server, and as shown in fig. 2, the storage scheduling server interacts with an application layer, a first database, and a second database, respectively, so as to receive data from the application layer and decide to store the data in one or both of the first database and the second database. The application layer is a data source for generating data, including but not limited to: an application layer for generating log data, an application layer for generating configuration data, and an application layer for generating service data.
As can be seen from the interaction diagram shown in fig. 2, the received data in step 101 is the data received by the storage scheduling server from the application layer.
When the first database is selected, the database which accords with the service characteristics can be selected according to the service characteristics. However, when selecting the second database, not only the service characteristics but also the estimated data volume of the service need to be considered. For example, the second database is distributed storage and is composed of a plurality of storage servers, and at this time, if the estimated data volume of the service is large, more storage servers can be selected to form a large distributed storage system; if the estimated data volume of the service is small, fewer storage servers can be selected to form a small distributed storage system.
The query performance is used for evaluating the response speed of the database to the query, and the quantitative representation mainly comprises two types: positive parameters and negative parameters, the positive parameters and the query performance are in positive relation, namely: the larger the forward parameter is, the higher the query performance is; smaller forward parameters represent lower query performance. For example, the forward parameter may be a response speed, and if the response speed is higher, it represents that the query performance is higher; the lower the response speed, the lower the query performance. And the negative parameter has a negative relation with the query performance, namely: the larger the negative parameter is, the lower the query performance is; a smaller negative going parameter represents a higher query performance. For example, the negative parameter may be a response duration, and if the response duration is shorter, the query performance is higher; the longer the response time, the lower the performance of the representative query. Of course, the response speed and the response duration are only the most commonly used quantitative representation of the query performance, and other parameters may also be used to represent the query performance in practical applications. The query performance of the first database and the second database selected by the embodiment of the invention has the following characteristics: the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the first database is lower than that of the second database after the accumulated amount of the received data reaches the preset data amount threshold.
In practical application, it is generally required to ensure that the database has a fast response speed and a short response time to the query, so as to improve the user experience. The present invention thus determines which database to store data to based on the monitored query performance.
Specifically, the response speed may be a data amount returned by the database for each query in a unit time, and for each query, a time difference between the receipt of the query and the return of the response data by the database and a data amount of the response data may be counted, and a ratio of the data amount of the response data for each query to the time difference for each query is used as the response speed for each query, and finally an average value of the response speeds for all queries is used as a final response speed. For example, if the database receives the query Q, Q at the time ST, returns the response data to the query Q, ET at the time ET, and the data amount of the response data to the query Q, Q is DN, and DN, respectively, the time difference between the query Q, and Q is ET-ST, and ET-ST, respectively, so as to obtain the response speed of the query Q, and Q is DN/(ET-ST), and DN/(ET-ST), respectively, and further obtain the final response speed of [ DN/(ET-ST) + DN/(ET-ST) ]/10. For the two response speeds of 100 MB/sec and 10 MB/sec calculated by the above method, it can be seen that the response speed of 10 MB/sec represents a lower query performance than that of 100 MB/sec.
The response time length may be an average time length that the database needs to consume for returning response data to the query, and for each query, the time difference between the receipt of the query and the return of the response data by the database may be counted, and an average value of the time differences of all queries is used as a final response time length. For example, if the database receives the queries Q1, Q2, … and Q10 at times ST1, ST2, … and ST10, respectively, and returns the response data to the queries Q1, Q2, … and Q10 at times ET1, ET2, … and ET10, respectively, the time difference between the queries Q1, Q2, … and Q10 is ET1-ST1, ET2-ST2, … and ET10-ST10, respectively, so as to obtain the response time length [ (ET1-ST1) + (ET2-ST2) + … + (ET10-ST10) ]/10. For the two response durations 10 ms and 50 ms calculated by the above method, it can be seen that the response duration 10 ms represents a higher query performance than the response duration 50 ms represents.
103, if the query performance of the first database is higher than that of the second database, or the query performance of the first database is the same as that of the second database, storing the data into the first database and the second database; the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the second database is higher than that of the first database after the accumulated amount of the received data reaches the preset data amount threshold.
In the embodiment of the present invention, the first database and the second database do not store any data in the initial state, that is, the accumulated amount of the received data is smaller than the preset data amount threshold, when the data is received, the query performance of the first database is higher than that of the second database, and at this time, the data may be stored in the first database and the second database, so that the two databases store the same data. With the continuous reception of data, the data amount in the first database and the second database is increased, so that the query performance of the first database and the query performance of the second database are both reduced, but the reduction speed of the query performance of the first database selected by the invention is higher than that of the second database, namely the query performance of the first database is reduced faster than that of the second database, and finally, the total data amount stored in the two databases, namely the accumulated amount of the received data, is the same when the total data amount reaches a preset data amount threshold value.
L1 is a curve showing the relationship between the response speed of the first database and the accumulated amount of the received data, L2 is a curve showing the relationship between the response speed of the second database and the accumulated amount of the received data, as can be seen from FIG. 3, the response speed of the first database is 100 MB/ms greater than the response speed of the second database is 80 MB/ms in the initial state, i.e., the query performance of the first database is higher than the query performance of the second database, as the accumulated amount of the received data increases, the response speed of the first database and the response speed of the second database both decrease, i.e., the query performance of the first database and the query performance of the second database both decrease, but the decrease speed of L1 is faster than the decrease speed of L2, i.e., the response speed of the first database decreases than the query performance of the second database, i.e., the response speed of the first database decreases, i.e., the response speed of the second database decreases, i.e., the response speed of the first database decreases than the response speed of the second database decreases, i.e., the response speed of the query performance of the second database decreases, and the response speed of the second database decreases, and thereafter, the response speed of the query performance of the second database decreases, i.e., the response speed decreases, when the query performance of the first database decreases, the query performance of the second database increases, the second database decreases, the second database increases, and the query performance decreases, the response speed decreases, the query performance decreases, the response speed of the second database decreases, the query performance decreases, the.
L3 is a graph of the relationship between the response time duration of the first database and the cumulative amount of the received data, L4 is a graph of the relationship between the response time duration of the second database and the cumulative amount of the received data, as seen in FIG. 4, the response time duration of the first database is less than 80 milliseconds of the response time duration of the second database in the initial state, i.e., the query performance of the first database is higher than the query performance of the second database, as the cumulative amount of the received data increases, the response time duration of the first database and the response time duration of the second database both increase, i.e., the query performance of the first database and the query performance of the second database both decrease, but the increase rate of the curve L3 is faster than the increase rate of the curve L4, i.e., the increase rate of the response time duration of the first database is greater than the increase rate of the response time duration of the second database, i.e., the response time duration of the query performance of the first database and the query performance of the second database both decrease, i.e., the response time duration of the query performance of the first database is greater than the query performance of the second database when the query performance of the second database increases, i.e., the query performance of the second database decreases, and the query performance of the second database decreases, i.e., the query performance of the second database decreases, and the query performance of the second database decreases, and the query performance of.
In the embodiment of the invention, before the accumulated amount of the received data reaches the preset data amount threshold, the received data is stored in the first database and the second database, and the same data is stored in the first database and the second database. At this point, the query may access the first database or the second database. And stopping storing the received data into the first database and only storing the received data into the second database after the accumulated amount of the received data reaches a preset data amount threshold DV corresponding to the P point.
In practical applications, in order to more accurately determine the performance gap between the first database and the second database, the query performance in the recent time period before the current time may be used as a reference. For example, if the time of receiving the data is 2019, 10, 9, 13:00, the query performance of the first database and the second database between 2019, 10, 5, 13:00 and 2019, 10, 9, 13:00 can be monitored respectively, and the accuracy is higher compared with the query performance of the first database and the second database between 2019, 9, 5, 13:00 and 2019, 9, 13: 00. It can be appreciated that queries closer to the current time are more accurate.
And 104, stopping storing the data in the first database if the query performance of the second database is higher than that of the first database.
And after the accumulated amount of the received data reaches a preset data amount threshold value, the query performance of the second database is higher than that of the first database, the data is stopped from being stored in the first database, and only the received data is stored in the second database. At this time, the query may access the second database, which may help to improve the query performance because the query performance of the second database is better.
The embodiment of the invention can be applied to a scene that future business volume cannot be predicted, the data can be stored in the first database and the second database at the initial stage of business, and along with the increase of business, the data can be stopped being stored in the first database with poor query performance and only stored in the second database with good query performance. Therefore, labor cost caused by the fact that the first database cannot meet the requirements of data volume and query performance in the later period and the data needs to be migrated to the second database can be reduced.
Of course, if it can be obviously predicted that the data volume of future services is large, only the second database can be directly used, and the data is always stored only in the second database; if it is obvious that the data volume of the future service is less, only the first database can be directly used, and the data is always stored only in the first database.
In practical application, two databases based on the invention can be expanded, and a third database can be selected, which has better query performance than the second database for large data volume storage, so that: at the initial stage of service, storing data into a first database, a second database and a third database, monitoring the query performance of the first database, the second database and the third database to obtain a first performance parameter, a second performance parameter and a third performance parameter, wherein the second performance parameter is greater than the first performance parameter along with the increase of data volume, and at the moment, only storing the data into the second database and the third database, and deleting the data in the first database; and as the data amount continues to increase, the third performance parameter is larger than the second performance parameter, and at the moment, only the data is stored in the third database, and the data in the second database is deleted. Thus, the database can be continuously expanded.
In summary, an embodiment of the present invention provides a data storage method, where the method includes: receiving data; second embodiment for separately monitoring query performance of first database and second database
Referring to fig. 5, a flowchart illustrating steps of another data storage method provided in an embodiment of the present invention is shown, which specifically includes the following steps:
This step can refer to the detailed description of step 101, and is not described herein again.
When the query performance of the second database is higher than that of the first database, the target database stops storing the data in the first database, only stores the data in the second database, so that the second database can be recorded as the target database, and if the target database exists, the data is directly stored in the target database. Therefore, the query performance of the first database and the query performance of the second database do not need to be continuously monitored, the computer resources consumed by monitoring are reduced, the time consumed by judging according to the query performance of the first database and the query performance of the second database can be saved, and the data storage speed is improved.
The embodiment of the invention can also determine which database the data is stored into by monitoring the query performance of the first database and the query performance of the second database when the target database does not exist.
The step 205 can refer to the detailed description of the step 103, and is not described herein again.
Optionally, the first database is an RDBMS database, and the second database is an O L AP database.
Among them, the RDBMS (Relational Database Management System) is mainly used for data storage and Management of small businesses, such as SQ L (Structured Query L language) Database System and ORAC L E (oracle) Database System, and for large businesses, the Query performance deteriorates rapidly due to the large amount of data.
The O L AP (On-L ine Analytical Processing) database is used for storing and Processing mass data, and has high query performance when storing the mass data.
According to the embodiment of the invention, the typical RDBMS database and the O L AP database are used as the first database and the second database to form a storage system, so that the query performance of the storage system is improved.
At step 206, an accumulated amount of received data is determined.
Specifically, the sum of the size of the data of this time and the accumulated amount of the received data recorded last time may be used as the updated accumulated amount of the received data each time the data is received; the size of the storage area occupied by the storage in the first database or the second database can also be directly used as the accumulated amount of the received data.
To further improve accuracy, it may be further determined whether to stop storing data into the first database based on the cumulative amount of received data and a data amount threshold. The data amount threshold may be empirically estimated, and may be, for example, DV in fig. 3. When the accumulated amount of the received data is small, the query performance of the first database is lower than that of the second database due to network abnormality, equipment abnormality and the like of the first database, and the situation is abnormal.
The embodiment of the invention can simultaneously judge by combining the data volume and the query performance, and can accurately stop storing data into the first database.
And step 208, taking the second database as a target database.
As can be seen from fig. 3 or 4, with the increasing of the data amount, after the accumulated amount of the received data reaches the preset data amount threshold DV at point P, the query performance of the second database is always higher than that of the first database, and at this time, the second database may be used as a target database to directly store the subsequently received data into the target database, and it is no longer necessary to monitor the query performance of the first database and the second database, and determine which database to store into.
Of course, if the target database does not exist, the accumulated amount of the received data does not reach the preset data amount threshold, and at this time, the query performance of the first database and the query performance of the second database need to be monitored to accurately determine which database the data is stored in.
The embodiment of the invention can reduce the time consumed by judgment and is beneficial to improving the storage speed.
The storage resource refers to a storage space of the first database.
The embodiment of the invention can recover the storage resource of the first database after the data is not stored in the first database any more, namely: the data and the data table stored in the first database are deleted, and the storage resources of the first database can be reused to store other data, so that the waste of the storage resources is avoided, and the utilization rate of the storage resources is improved.
In summary, an embodiment of the present invention provides a data storage method, where the method includes: receiving data; respectively monitoring the query performance of the first database and the query performance of the second database; if the query performance of the first database is higher than that of the second database, or the query performance of the first database is the same as that of the second database, storing the data into the first database and the second database; the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the second database is higher than that of the first database after the accumulated amount of the received data reaches the preset data amount threshold; and if the query performance of the second database is higher than that of the first database, stopping storing the data in the first database. The database is selected according to the query performance, the technical problem that the database cannot adapt to the data storage requirement in the later period, so that the query performance of the database is poor can be solved, and the technical effect of improving the query performance of the database can be achieved.
EXAMPLE III
Referring to fig. 6, a flowchart illustrating steps of a storage scheduling server according to an embodiment of the present invention is shown, which specifically includes the following steps:
a data receiving module 301, configured to receive data.
A performance monitoring module 302, configured to monitor query performance of the first data and query performance of the second data, respectively.
A first storage module 303, configured to store the data in the first database and the second database if the query performance of the first database is higher than the query performance of the second database, or if the query performance of the first database is the same as the query performance of the second database; the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the second database is higher than that of the first database after the accumulated amount of the received data reaches the preset data amount threshold.
A second storing module 304, configured to stop storing the data in the first database if the query performance of the second database is higher than the query performance of the first database.
To sum up, an embodiment of the present invention provides a storage scheduling server, where the server includes: the data receiving module is used for receiving data; the performance monitoring module is used for respectively monitoring the query performance of the first data and the query performance of the second data; the first storage module is used for storing the data into the first database and the second database if the query performance of the first database is higher than that of the second database or the query performance of the first database is the same as that of the second database; the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the second database is higher than that of the first database after the accumulated amount of the received data reaches the preset data amount threshold; and the second storage module is used for stopping storing the data to the first database if the query performance of the second database is higher than that of the first database. The database is selected according to the query performance, the technical problem that the database cannot adapt to the data storage requirement in the later period, so that the query performance of the database is poor can be solved, and the technical effect of improving the query performance of the database can be achieved.
Example four
Referring to fig. 7, a flowchart illustrating steps of another storage scheduling server according to an embodiment of the present invention is shown, which specifically includes the following steps:
a data receiving module 401, configured to receive data.
A third storing module 402, configured to store the data in a target database if the target database exists.
A performance monitoring entering module 403, configured to enter the performance monitoring module 404 if the target database does not exist.
And a performance monitoring module 404, configured to monitor query performance of the first data and query performance of the second data, respectively.
A first storage module 405, configured to store the data in the first database and the second database if the query performance of the first database is higher than the query performance of the second database, or the query performance of the first database is the same as the query performance of the second database; the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the second database is higher than that of the first database after the accumulated amount of the received data reaches the preset data amount threshold.
A cumulative amount determination module 406 to determine a cumulative amount of received data.
A second storing module 407, configured to stop storing the data in the first database if the query performance of the second database is higher than the query performance of the first database and the accumulated amount is greater than or equal to a preset data amount threshold.
A target database determination module 408, configured to use the second database as a target database.
And the resource recycling module 409 is used for recycling the storage resources of the first database.
Optionally, in another embodiment of the present invention, the first database is an RDBMS database, and the second database is an O L AP database.
To sum up, an embodiment of the present invention provides a storage scheduling server, where the server includes: the data receiving module is used for receiving data; the performance monitoring module is used for respectively monitoring the query performance of the first data and the query performance of the second data; the first storage module is used for storing the data into the first database and the second database if the query performance of the first database is higher than that of the second database or the query performance of the first database is the same as that of the second database; the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the second database is higher than that of the first database after the accumulated amount of the received data reaches the preset data amount threshold; and the second storage module is used for stopping storing the data to the first database if the query performance of the second database is higher than that of the first database. The database is selected according to the query performance, the technical problem that the database cannot adapt to the data storage requirement in the later period, so that the query performance of the database is poor can be solved, and the technical effect of improving the query performance of the database can be achieved.
An embodiment of the present invention further provides an electronic device, as shown in fig. 8, which includes a processor 501, a communication interface 502, a memory 503 and a communication bus 504, where the processor 501, the communication interface 502 and the memory 503 complete mutual communication through the communication bus 504,
a memory 503 for storing a computer program;
the processor 501, when executing the program stored in the memory 503, implements the following steps:
receiving data;
respectively monitoring the query performance of the first database and the query performance of the second database;
if the query performance of the first database is higher than that of the second database, or the query performance of the first database is the same as that of the second database, storing the data into the first database and the second database; the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the second database is higher than that of the first database after the accumulated amount of the received data reaches the preset data amount threshold;
and if the query performance of the second database is higher than that of the first database, stopping storing the data in the first database.
The communication bus mentioned in the above terminal may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the terminal and other equipment.
The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
In yet another embodiment of the present invention, a computer-readable storage medium is further provided, which has instructions stored therein, and when the instructions are executed on a computer, the computer is caused to execute the data storage method described in any one of the above embodiments.
In a further embodiment of the present invention, there is also provided a computer program product containing instructions which, when run on a computer, cause the computer to perform the data storage method of any of the above embodiments.
The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, e.g., from one website site, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DS L)) or wireless (e.g., infrared, wireless, microwave, etc.) manner to another website site, computer, server, or data center.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and for the relevant points, reference may be made to the partial description of the method embodiment.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (12)
1. A data storage method is applied to a storage scheduling server, and comprises the following steps:
receiving data;
respectively monitoring the query performance of the first database and the query performance of the second database;
if the query performance of the first database is higher than that of the second database, or the query performance of the first database is the same as that of the second database, storing the data into the first database and the second database; the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the second database is higher than that of the first database after the accumulated amount of the received data reaches the preset data amount threshold;
and if the query performance of the second database is higher than that of the first database, stopping storing the data in the first database.
2. The method of claim 1, wherein prior to monitoring the query performance of the first database and the query performance of the second database, respectively, the method further comprises:
if the target database exists, storing the data into the target database;
if the target database does not exist, the step of respectively monitoring the query performance of the first database and the query performance of the second database is carried out;
after the ceasing to store the data to the first database, the method further comprises:
and taking the second database as a target database.
3. The method of claim 1, wherein after the receiving data, the method further comprises:
determining an accumulated amount of received data;
if the query performance of the second database is higher than the query performance of the first database, stopping storing the data in the first database, including:
and if the query performance of the second database is higher than that of the first database and the accumulated amount is greater than or equal to a preset data amount threshold value, stopping storing the data in the first database.
4. The method of any one of claims 1 to 3, wherein after stopping storing the data to the first database, further comprising:
and recovering the storage resources of the first database.
5. The method of claim 4, wherein the first database is an RDBMS database and the second database is an O L AP database.
6. A storage scheduling server, the server comprising:
the data receiving module is used for receiving data;
the performance monitoring module is used for respectively monitoring the query performance of the first data and the query performance of the second data;
the first storage module is used for storing the data into the first database and the second database if the query performance of the first database is higher than that of the second database or the query performance of the first database is the same as that of the second database; the query performance of the first database is higher than that of the second database before the accumulated amount of the received data reaches a preset data amount threshold, and the query performance of the second database is higher than that of the first database after the accumulated amount of the received data reaches the preset data amount threshold;
and the second storage module is used for stopping storing the data to the first database if the query performance of the second database is higher than that of the first database.
7. The server of claim 6, further comprising:
the third storage module is used for storing the data into a target database if the target database exists;
the performance monitoring entering module is used for entering the performance monitoring module if the target database does not exist;
and the target database determining module is used for taking the second database as a target database.
8. The server of claim 6, further comprising:
the accumulated amount determining module is used for determining the accumulated amount of the received data;
the second storage module is further configured to:
and if the query performance of the second database is higher than that of the first database and the accumulated amount is greater than or equal to a preset data amount threshold value, stopping storing the data in the first database.
9. The server according to any one of claims 6 to 8, wherein the server further comprises:
and the resource recovery module is used for recovering the storage resources of the first database.
10. The server according to claim 9, wherein the first database is an RDBMS database and the second database is an O L AP database.
11. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 5 when executing a program stored in the memory.
12. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010220039.XA CN111444184A (en) | 2020-03-25 | 2020-03-25 | Data storage method, storage scheduling server, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010220039.XA CN111444184A (en) | 2020-03-25 | 2020-03-25 | Data storage method, storage scheduling server, electronic device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111444184A true CN111444184A (en) | 2020-07-24 |
Family
ID=71650721
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010220039.XA Pending CN111444184A (en) | 2020-03-25 | 2020-03-25 | Data storage method, storage scheduling server, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111444184A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113760889A (en) * | 2020-11-09 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Data processing method, device, computer system and computer readable storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104246728A (en) * | 2012-04-27 | 2014-12-24 | 英派尔科技开发有限公司 | Multiple variable coverage memory for database indexing |
| US20170124148A1 (en) * | 2015-10-29 | 2017-05-04 | International Business Machines Corporation | Index table based routing for query resource optimization |
| CN107622091A (en) * | 2017-08-23 | 2018-01-23 | 阿里巴巴集团控股有限公司 | A kind of data base query method and device |
| CN109408496A (en) * | 2018-09-18 | 2019-03-01 | 沈文策 | A kind of method and device reducing data redundancy |
| CN109710675A (en) * | 2018-12-26 | 2019-05-03 | 深圳乐信软件技术有限公司 | A kind of storing data library switching method, device, server and storage medium |
| CN110389967A (en) * | 2019-07-26 | 2019-10-29 | 深圳市腾讯计算机系统有限公司 | Date storage method, device, server and storage medium |
-
2020
- 2020-03-25 CN CN202010220039.XA patent/CN111444184A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104246728A (en) * | 2012-04-27 | 2014-12-24 | 英派尔科技开发有限公司 | Multiple variable coverage memory for database indexing |
| US20170124148A1 (en) * | 2015-10-29 | 2017-05-04 | International Business Machines Corporation | Index table based routing for query resource optimization |
| CN107622091A (en) * | 2017-08-23 | 2018-01-23 | 阿里巴巴集团控股有限公司 | A kind of data base query method and device |
| CN109408496A (en) * | 2018-09-18 | 2019-03-01 | 沈文策 | A kind of method and device reducing data redundancy |
| CN109710675A (en) * | 2018-12-26 | 2019-05-03 | 深圳乐信软件技术有限公司 | A kind of storing data library switching method, device, server and storage medium |
| CN110389967A (en) * | 2019-07-26 | 2019-10-29 | 深圳市腾讯计算机系统有限公司 | Date storage method, device, server and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113760889A (en) * | 2020-11-09 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Data processing method, device, computer system and computer readable storage medium |
| CN113760889B (en) * | 2020-11-09 | 2024-05-17 | 北京沃东天骏信息技术有限公司 | Data processing method, device, computer system and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107301215B (en) | Search result caching method and device and search method and device | |
| CN112087644B (en) | Pull stream request processing method, device and system, electronic equipment and storage medium | |
| WO2021227677A1 (en) | Scene pushing method, scene executing method, scene pushing device, scene executing device, terminal, server, and system | |
| CN113297183B (en) | Alarm analysis method and device for time window | |
| CN111782692B (en) | Frequency control method and device | |
| CN109450969B (en) | Method and device for acquiring data from third-party data source server and server | |
| CN101636000A (en) | Treating method and treatment device of alarm storms | |
| CN105512129A (en) | Method and device for mass data retrieval, and method and device for storing mass data | |
| CN108958883B (en) | Recovery method and system for virtual machine in cloud computing cluster | |
| CN111291252B (en) | Method and device for adjusting query rate per second, electronic equipment and storage medium | |
| CN113722106A (en) | Memory management method and device | |
| CN110311975B (en) | Data request processing method and device | |
| CN111444184A (en) | Data storage method, storage scheduling server, electronic device and storage medium | |
| CN108829735B (en) | Synchronization method, device, server and storage medium for parallel execution plan | |
| CN114090268B (en) | Container management method and container management system | |
| CN110543509A (en) | Monitoring system, method and device for user access data and electronic equipment | |
| CN115774602A (en) | Container resource allocation method, device, equipment and storage medium | |
| CN113656178B (en) | Data processing method, device, equipment and readable storage medium | |
| CN114553944A (en) | Early warning message pushing method and system | |
| CN110113187B (en) | Configuration updating method and device, configuration server and configuration system | |
| CN113835905A (en) | Message queue load balancing method and device, electronic equipment and medium | |
| CN112597354A (en) | Method, device, system and storage medium for providing configuration parameters | |
| CN112148508A (en) | Information processing method and related device | |
| CN113760398A (en) | Interface calling method, server, system and storage medium | |
| CN112015924A (en) | Streaming media caching method and device and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |