CN115269558A - Data storage method, apparatus, device, storage medium and program product - Google Patents
Data storage method, apparatus, device, storage medium and program product Download PDFInfo
- Publication number
- CN115269558A CN115269558A CN202210982516.5A CN202210982516A CN115269558A CN 115269558 A CN115269558 A CN 115269558A CN 202210982516 A CN202210982516 A CN 202210982516A CN 115269558 A CN115269558 A CN 115269558A
- Authority
- CN
- China
- Prior art keywords
- data
- stored
- database
- target database
- processing instruction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data storage method, a data storage device, a computer device, a storage medium and a computer program product. The method can be used in the field of financial technology or other related fields, and comprises the following steps: acquiring the data storage capacity of data to be stored; acquiring storage space information of a target database for storing data to be stored; obtaining a database processing instruction aiming at a target database based on the data storage capacity and the storage space information; and sending the database processing instruction to a target database so that the target database stores the data to be stored according to the database processing instruction. By adopting the method, data can be efficiently stored.
Description
Technical Field
The present application relates to the field of data migration technologies, and in particular, to a data storage method, an apparatus, a computer device, a storage medium, and a computer program product.
Background
With the development of the technical field of data migration, a data migration storage technology from a centralized database to a distributed database appears, and the technical scheme migrates and stores data from the centralized database to the distributed database in a preset data migration mode.
In the above technical solution, the memory occupied by the data in the migration process may be increased, which may make the memory of the distributed database insufficient to store the migrated data, in the prior art, generally, after the data migration, the memory usage rate of the distributed database is monitored, and if the memory usage rate is greater than a threshold, the distributed database needs to be manually cleaned and expanded. However, this results in inefficient data migration storage.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a data storage method, an apparatus, a computer device, a computer readable storage medium, and a computer program product, which can efficiently perform data migration and storage.
In a first aspect, the present application provides a data storage method. The method comprises the following steps:
acquiring the data storage capacity of data to be stored;
acquiring storage space information of a target database for storing the data to be stored;
obtaining a database processing instruction aiming at the target database based on the data storage amount and the storage space information;
and sending the database processing instruction to the target database so that the target database stores the data to be stored according to the database processing instruction.
In one embodiment, the number of the data to be stored is multiple; the method for acquiring the data storage capacity of the data to be stored comprises the following steps: acquiring the quantity of the data to be stored and the average data storage quantity of each data to be stored; and obtaining the data storage quantity of the data to be stored based on the quantity of the data to be stored and the average data storage quantity.
In one of them in an embodiment of the present invention, the acquiring the quantity of the data to be stored includes: acquiring the quantity of original data; the original data is data stored in an original database, and the data to be stored is obtained by converting the original data; acquiring data conversion information for converting the original data into the data to be stored; and obtaining the quantity of the data to be stored based on the data conversion information and the quantity of the original data.
In one embodiment, the obtaining an average data storage amount of each piece of data to be stored includes: obtaining the storage amount of the stored data in the target database and obtaining the quantity of the stored data; the data category of the stored data is the same as that of the data to be stored, and the number of the stored data is multiple; and obtaining the average data storage quantity of each data to be stored based on the quantity of the stored data and the storage quantity of the stored data.
In one embodiment, the obtaining of the database processing instruction for the target database based on the data storage amount and the storage space information includes: obtaining the estimated utilization rate of the target database based on the data storage capacity and the storage space information; the estimated utilization rate is used for representing the utilization rate of a storage space of the target database after the data to be stored is stored; setting the database processing instruction as a first instruction for increasing the storage space of the target database under the condition that the estimated utilization rate is greater than or equal to a preset threshold value; under the condition that the estimated utilization rate is smaller than the preset threshold value, setting the database processing instruction as a second instruction for storing the data to be stored; sending the database processing instruction to the target database to enable the target database to store the data to be stored according to the database processing instruction, wherein the method comprises the following steps: under the condition that the database processing instruction is a first instruction, sending the database processing instruction to the target database so that the target database performs augmentation processing on the storage space; if the estimated utilization rate is smaller than the preset threshold value after the storage space of the target database is increased, setting the database processing instruction as the second instruction; and under the condition that the database processing instruction is a second instruction, sending the database processing instruction to the target database so as to enable the target database to store the data to be stored.
In one embodiment, after the sending the database processing instruction to the target database to make the target database perform the increase processing on the storage space if the database processing instruction is the first instruction, the method further includes: if the estimated utilization rate is larger than or equal to the preset threshold value after the storage space of the target database is increased, setting the database processing instruction as a third instruction for storing the data to be stored to a big data platform; sending the database processing instruction to the target database so that the target database stores the data to be stored according to the database processing instruction, and further comprising: and under the condition that the database processing instruction is a third instruction, sending the database processing instruction to the target database, so that the target database sets the database processing instruction as the data to be stored in a big data platform.
In a second aspect, the present application further provides a data storage device. The device comprises:
the data storage quantity acquisition module is used for acquiring the data storage quantity of the data to be stored;
the storage space information acquisition module is used for acquiring the storage space information of a target database for storing the data to be stored;
the processing instruction acquisition module is used for acquiring a database processing instruction aiming at the target database based on the data storage amount and the storage space information;
and the processing instruction execution module is used for sending the database processing instruction to the target database so as to enable the target database to store the data to be stored according to the database processing instruction.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the following steps when executing the computer program:
acquiring the data storage capacity of data to be stored;
acquiring storage space information of a target database for storing the data to be stored;
obtaining a database processing instruction aiming at the target database based on the data storage amount and the storage space information;
and sending the database processing instruction to the target database so that the target database stores the data to be stored according to the database processing instruction. In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium, on which a computer program is stored which, when executed by a processor, carries out the steps of:
acquiring the data storage capacity of data to be stored;
acquiring storage space information of a target database for storing the data to be stored;
obtaining a database processing instruction aiming at the target database based on the data storage amount and the storage space information;
and sending the database processing instruction to the target database so that the target database stores the data to be stored according to the database processing instruction.
In a fifth aspect, the present application further provides a computer program product. The computer program product comprising a computer program which when executed by a processor performs the steps of:
acquiring the data storage capacity of data to be stored;
acquiring storage space information of a target database for storing the data to be stored;
obtaining a database processing instruction aiming at the target database based on the data storage amount and the storage space information;
and sending the database processing instruction to the target database so that the target database stores the data to be stored according to the database processing instruction.
According to the data storage method, the data storage device, the computer equipment, the storage medium and the computer program product, the data storage quantity of the data to be stored is obtained; acquiring storage space information of a target database for storing data to be stored; obtaining a database processing instruction aiming at a target database based on the data storage capacity and the storage space information; and sending the database processing instruction to a target database so that the target database stores the data to be stored according to the database processing instruction. According to the data storage method and device, the data storage quantity of the data to be stored is obtained through pre-estimation, and then the processing instruction aiming at the target database is obtained according to the data storage quantity and the storage space information of the target database, so that the target database stores the data to be stored according to the processing instruction, and the data storage can be efficiently carried out.
Drawings
FIG. 1 is a diagram of an exemplary data storage system;
FIG. 2 is a schematic flow chart diagram illustrating a data storage method according to one embodiment;
FIG. 3 is a flow chart illustrating a process of obtaining a data storage amount of data to be stored in one embodiment;
FIG. 4 is a flow diagram illustrating the process of obtaining the amount of data to be stored in one embodiment;
FIG. 5 is a flowchart illustrating obtaining an average data storage amount of each data to be stored according to an embodiment;
FIG. 6 is a diagram illustrating an exemplary implementation of a data migration storage system;
FIG. 7 is a flowchart illustrating a data migration storage method according to an embodiment;
FIG. 8 is a block diagram of a data storage device in one embodiment;
FIG. 9 is a diagram illustrating an internal structure of a computer device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.
It should be noted that the terms "first \ second \ third" related to the embodiments of the present invention only distinguish similar objects, and do not represent a specific ordering for the objects, and it should be understood that "first \ second \ third" may exchange a specific order or sequence when allowed. It should be understood that the terms first, second, and third, as used herein, are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in other sequences than those illustrated or otherwise described herein.
The data storage method provided by the embodiment of the application can be applied to the application environment shown in fig. 1. Wherein, the analysis end 101 communicates with the target database 102 through a network. The data storage system may store data that the target database 102 needs to process. The data storage system may be integrated on the target database 102, or may be located on the cloud or other network server. The analysis end 101 acquires the data storage capacity of the data to be stored; acquiring storage space information of a target database 102 for storing data to be stored; the analysis end 101 obtains a database processing instruction for the target database 102 based on the data storage amount and the storage space information; the database processing instruction is sent to the target database 102, so that the target database 102 stores the data to be stored according to the database processing instruction. The analysis end 101 and the target database 102 may be implemented by separate servers or a server cluster composed of a plurality of servers.
In one embodiment, as shown in fig. 2, a data storage method is provided, which is described by taking the method as an example applied to the analysis end 101 in fig. 1, and includes the following steps:
step S201, acquiring a data storage amount of data to be stored.
When the data to be stored is the centralized database and the original data is migrated to the distributed database for storage, the original data is converted according to a preset conversion rule and is stored in the distributed database, and the data storage amount is the data size of the data to be stored.
Specifically, the data volume of original data to be migrated in the centralized database is obtained according to a preset data migration plan, and then the data storage volume of the data to be stored is estimated according to a preset migration data conversion rule based on the data volume of the original data.
Step S202, obtaining storage space information of the target database 102 for storing data to be stored.
The target database 102 is a distributed database, the database is used for storing data migrated from the centralized database, and the storage space information refers to the remaining storage space of the target database 102.
Specifically, the amount of data stored in the target database 102 is obtained, and then the remaining storage space of the target database 102 is calculated according to the total amount of storage in the target database 102.
In step S203, a database processing instruction for the target database 102 is obtained based on the data storage amount and the storage space information.
The database processing instruction refers to an instruction for processing the database so that data to be stored can be stored, for example, the data processing instruction may be to perform defragmentation on a data table with a high fragmentation rate of the target database 102 to release space; or adjusting the data retention period to the minimum retention period and performing the cleaning of the data table.
Specifically, it is estimated whether the data storage amount exceeds the remaining storage space of the target database 102 according to the estimated data storage amount and the remaining storage space of the target database 102, and if the data storage amount exceeds the remaining storage space of the target database 102, a corresponding database processing instruction is generated to clean or expand the remaining storage space of the target database 102, so that the data to be stored can be stored.
Step S204, sending the database processing instruction to the target database 102, so that the target database 102 stores the data to be stored according to the database processing instruction.
Specifically, the corresponding database processing instruction is sent to the target database 102, and the target database 102 cleans up the database storage space or expands the database storage space according to the database processing instruction, so that the data to be stored is stored.
In the data storage method, the data storage quantity of the data to be stored is obtained; acquiring storage space information of a target database 102 for storing data to be stored; obtaining a database processing instruction for the target database 102 based on the data storage amount and the storage space information; the database processing instruction is sent to the target database 102, so that the target database 102 stores the data to be stored according to the database processing instruction. According to the data storage method and device, the data storage quantity of the data to be stored is obtained through pre-estimation, and then the processing instruction for the target database 102 is obtained according to the data storage quantity and the storage space information of the target database 102, so that the target database 102 stores the data to be stored according to the processing instruction, and the data storage can be efficiently carried out.
In one embodiment, as shown in fig. 3, acquiring the data storage amount of the data to be stored includes the following steps:
step S301, acquiring the number of data to be stored, and acquiring the average data storage amount of each data to be stored.
The number of the data to be stored is the number of data pieces of the data to be stored, for example, the data to be stored includes ten thousand pieces of data, and the average data storage amount refers to an average value of the data storage amounts of each piece of data in the data to be stored.
Specifically, the number of data to be migrated is obtained based on a preset data migration plan, the number of data to be stored is further obtained, and then the average data storage amount of the historical data is used as the average data storage amount of each piece of data to be stored.
Step S302, based on the number of the data to be stored and the average data storage amount, obtaining the data storage amount of the data to be stored.
Specifically, the data storage amount of the data to be stored can be calculated according to the number of the data to be stored and the average data storage amount.
In this embodiment, the number of the data to be stored is obtained based on the preset data migration plan, and the average data storage amount of each data to be stored is obtained, so that the data storage amount of the data to be stored can be accurately estimated.
In one embodiment, as shown in fig. 4, obtaining the amount of data to be stored includes the following steps:
step S401, acquiring the quantity of original data; the original data are data stored in an original database, and the data to be stored are obtained by converting the original data.
The original data is data to be migrated to the target database 102 in the original database, and the original database is a centralized database for migrating the original data to the target database 102 for storage.
Specifically, the amount of the original data is obtained by a preset data migration plan.
Step S402, acquiring data conversion information for converting the original data into the data to be stored.
The data conversion information is preset, and when the data is migrated from the centralized database to the distributed database for storage, the original data is converted into the conversion rule of the data to be stored.
Specifically, a conversion rule of data to be stored that is set in advance is acquired.
Step S403, obtaining the amount of data to be stored based on the data conversion information and the amount of the original data.
Specifically, the number of data to be stored is calculated from the number of raw data based on a preset data conversion rule.
In this embodiment, the number of data to be stored can be accurately obtained through the preset data conversion information.
In one embodiment, as shown in fig. 5, obtaining the average data storage amount of each data to be stored includes the following steps:
step S501, the storage amount of the stored data in the target database 102 is obtained, and the quantity of the stored data is obtained; the stored data and the data to be stored have the same data type, and the number of the stored data is multiple.
The storage amount of the stored data is the data amount of the existing data in the target database 102, and the number of the stored data is the number of the existing data in the target database 102, where the stored data and the data to be stored are the same type of data, and the average storage amount of each piece of data is substantially the same.
Specifically, the storage amount of the stored data and the amount of the stored data are obtained from the target database 102 by the data detection tool.
Step S502, based on the number of the stored data and the storage amount of the stored data, an average data storage amount of each to-be-stored data is obtained.
Specifically, based on the number of stored data and the storage amount of the stored data, an average data storage amount of each stored data is obtained as an average data storage amount of each data to be stored.
In this embodiment, the average data storage amount of each to-be-stored data can be accurately obtained by acquiring the number of the stored data and the storage amount of the stored data.
In one embodiment, obtaining database processing instructions for the target database 102 based on the amount of data storage and the storage space information comprises the following steps:
obtaining the estimated utilization rate of the target database 102 based on the data storage capacity and the storage space information; the estimated utilization rate is used for representing the utilization rate of a storage space after the target database 102 stores data to be stored; setting the database processing instruction as a first instruction for increasing the storage space of the target database 102 when the estimated utilization rate is greater than or equal to a preset threshold; and under the condition that the estimated utilization rate is smaller than a preset threshold value, setting the database processing instruction as a second instruction for storing the data to be stored.
The estimated utilization rate is a ratio of a total amount of stored data after the target database 102 stores data to be stored to a total space size of the target database 102, the preset threshold is a preset value, the first instruction is an instruction when the estimated utilization rate is greater than or equal to the preset threshold, and the second instruction is an instruction when the estimated utilization rate is less than the preset threshold.
Specifically, based on the data storage amount and the storage space information, a ratio of a total amount of stored data after the target database 102 stores the data to be stored to a total space size of the target database 102 is calculated and obtained, and is used as an estimated utilization rate of the target database 102, and if the estimated utilization rate is greater than or equal to a preset threshold, a first instruction is generated: increasing the storage space of the target database 102; if the estimated utilization rate is smaller than a preset threshold value, generating a second instruction: the data to be stored is stored to the target database 102.
Sending the database processing instruction to the target database 102, so that the target database 102 stores the data to be stored according to the database processing instruction, including: under the condition that the database processing instruction is a first instruction, the database processing instruction is sent to the target database 102, so that the target database 102 performs increasing processing on the storage space; if the estimated utilization rate is smaller than the preset threshold value after the storage space of the target database 102 is increased, setting the database processing instruction as a second instruction; and in the case that the database processing instruction is the second instruction, sending the database processing instruction to the target database 102 so that the target database 102 stores the data to be stored.
Where the augmentation process augments the capacity of the target database 102, for example, a backup database server may be turned on, clearing some historical data of failures in the database.
Specifically, a first instruction is sent to the target database 102, so that the target database 102 performs an increase process on the storage space, for example, a standby database server is started, and some failed historical data in the database are cleared; a second instruction is sent to the target database 102 so that the target database 102 stores the data to be stored.
In this embodiment, the estimated utilization rate is obtained, and whether the estimated utilization rate exceeds a preset value is determined, so that a database processing instruction is generated to enable the target database 102 to perform corresponding processing, and the data to be stored can have enough space to be stored.
In one embodiment, in a case that the database processing instruction is the first instruction, after sending the database processing instruction to the target database 102 to enable the target database 102 to perform the increase processing on the storage space, the method further includes the following steps:
if the estimated utilization rate is greater than or equal to the preset threshold value after the storage space of the target database 102 is increased, setting the database processing instruction as a third instruction for storing the data to be stored to the big data platform; and under the condition that the database processing instruction is the third instruction, sending the database processing instruction to the target database 102, so that the target database 102 stores the database processing instruction which is set to store the data to be stored to the big data platform.
The big data platform is a network cloud disk, and the third instruction is a database processing instruction generated after the target database 102 completes the first instruction and when the estimated utilization rate is still greater than or equal to the preset threshold value.
Specifically, after the target database 102 completes the first instruction and increases the storage space of the target database 102, if the estimated utilization rate is still greater than or equal to the preset threshold value, a third instruction is generated, so that the data to be stored is stored in the big data platform.
In this embodiment, after the target database 102 completes the first instruction, the estimated utilization rate is still greater than or equal to the preset threshold, and a third instruction for storing the data to be stored to the big data platform is generated, so that the data to be stored can still be stored when the data to be stored cannot be stored by the target database 102.
In an application embodiment, as shown in fig. 6, a database storage space securing system (hereinafter referred to as data migration storage system) applied to system migration is provided, and includes a centralized application server, a centralized database, a file server, a distributed application server, a distributed database, a storage space analysis processing server, and a big data platform. The automatic evaluation of the storage space of the database is realized when the traditional centralized system is migrated to the novel distributed database system, and according to different conditions, the guarantee measures are automatically taken according to the priority which affects the system least, so that the problem of insufficient storage space is solved.
The centralized application server is a centralized mainframe or a centralized platform application server, and an application system operated by the centralized application server needs to be redesigned and migrated to the distributed application server; the centralized database is a database server corresponding to a centralized mainframe or a centralized platform application server and is used for storing parameters and data of the operation of the application server, and the data stored in the centralized database needs to be redesigned and migrated to the distributed database; the file server is a file system for temporarily storing data when the data stored in the centralized database is migrated, and is used for the centralized database to extract the data of the centralized database server and temporarily store the data, and files stored by the file server are read by the distributed application server and are recombined and stored in the distributed database according to a new data structure; the distributed application server is a target application system migrated by the centralized application system, is reconstructed according to a distributed system architecture according to a centralized application system application program, and comprises a PAAS cloud-based container and an IAAS cloud-based virtual machine or physical machine, and when the storage space of the database is insufficient, the distributed application server receives database adjustment information sent by the storage space analysis processing server and executes corresponding countermeasures, wherein the countermeasures comprise adjusting the retention period of a data table, starting a cleaning program, adjusting the access mode of the data table and the like; the distributed database is a target database system migrated by the centralized database system, and is generally a mysql, golden DB and other databases; the big data platform is used for backing up the data cleaned by the distributed database, and when the data of the distributed database is cleaned and the distributed application server needs to access the cleaned data, the big data platform provides data access service.
The storage space analysis processing server is an application server which is used for estimating, obtaining and trial-calculating storage space and taking database security measures under the condition of insufficient storage space, and comprises the following steps: the device comprises a storage space acquisition module, a tangent flow and rule configuration module, a storage space analysis module and a storage space guarantee module.
Wherein, the storage space acquisition module: the module is in butt joint with a distributed database, and obtains the utilization rate of the disk space of a database server and the occupation of the use space (including data space, index space, fragment space and the like) of each data table of the database by executing a database instruction and an operating system instruction.
A tangent flow and rule configuration module: providing two broad types of configurations including old and new system cutflows planning configuration and new and old system migration rule base. The new and old system tangent plan configuration comprises the date and time of each tangent of the new and old system and the scale and data of each tangent, for example, if the merchant acquiring system is migrated, 0 o' clock 0 of a certain day of a certain month of a first time can be set to be a tangent time window, the merchant list and number of the tangent are 10 ten thousand, the system transaction amount is 50 ten thousand, and the like, and the time and scale of the second, third, etc. current is set. For example, when the old system generates an online transaction, each record of the table a of the old system is 5KB, and the new system is migrated to a corresponding record of the table a 'and two records of the table B', and each record is estimated to be 2KB and 4KB respectively. By analogy, the comparison relations of all the table structures of the new system and the old system are set and the data table space is estimated.
In addition, the flow switching and rule configuration module also receives the analysis result of the storage space analysis module, and adjusts the flow switching parameters and the migration rule base, namely when the storage space analysis result is that the storage space is insufficient, the flow switching plan needs to be suspended or adjusted; and when the storage space analysis result shows that the actual storage space occupation does not accord with the estimation, updating the migration rule base into a new migration rule according to the data recalculated by the storage space analysis module.
A storage space analysis module: and acquiring the number of transaction strokes according to the real-time storage space condition acquired by the storage space acquisition module and the running condition of the new system, recalculating according to the rule base and the tangential flow plan data set by initialization, and analyzing and evaluating whether the initialized parameters are accurate or not. For example, the last cut flow is 100 ten thousand a table records, and according to the preset migration rule parameters, 100 ten thousand a ' table records and 200 ten thousand records of 100 ten thousand records and a B ' table are generated, and the occupied space of 1.9G of the a ' table and the occupied space of 3.8G of the B ' table are increased, but actually, because the variable length field is different from the estimated time, the space of 5G is actually increased in the B ' table.
The storage space analysis module is used for recalculating the storage space required by the next migration according to the migration transaction data volume and date set by the next tangential plan and the migration rule of the migration rule base before each tangential, calculating whether the storage space utilization rate exceeds a threshold value according to the storage space utilization rate of the current database and the storage space required by the next migration, and informing the storage space guarantee module to implement storage space guarantee measures if the storage space utilization rate exceeds the threshold value.
Storage space guarantee module: the module implements a space guarantee strategy according to a priority level according to a set data space guarantee measure. Firstly, executing a safeguard measure 1 (space expansion) to increase the available space of the database server by adjusting the space distributed by the storage space of the host where the database server is located; when the space of the host machine is insufficient, the guarantee measure 1 cannot be implemented, a guarantee measure 2 (fragment cleaning) is adopted, the data table with high database fragment rate is defragmented, the space is released, if the guarantee measure 2 is implemented, the guarantee threshold value of the use rate of the storage space of the database is not reached, a guarantee measure 3 is executed, the data retention period is adjusted, the minimum retention period is adjusted, and the cleaning of the data table is executed. And if the guarantee measure 3 is implemented, the guarantee threshold value of the storage space utilization rate of the database is not reached, and if the guarantee measure 4 (data migration), the data is migrated to the big data platform.
When it needs to be explained, the reason for setting the priority of the above safeguard measures is as follows: the guarantee measure 1 (space expansion) has minimum influence on the system, but depends on whether the host where the database server is located has enough space; a safeguard measure 2 (defragmentation) which has no influence on the original data of the database table, but influences the operation of online transactions during defragmentation; the guarantee measure 3 is to adjust the data retention period to the minimum retention period required to be retained by the data table, and the data exceeding the minimum retention period is the data which does not need to be accessed by the service system, but is cleaned after all, thereby being not beneficial to the technical analysis of some production problems; and 4, ensuring measures 4 (data migration), wherein the part of data needs to be accessed, but the access efficiency is reduced due to the migration to a heterogeneous large data platform.
In this embodiment, the data storage amount of the data to be stored is obtained by pre-estimation, and then the processing instruction for the target database is obtained according to the data storage amount and the storage space information of the target database, so that the target database stores the data to be stored according to the processing instruction, the data amount does not exceed the storage space of the database, and the data storage can be efficiently performed.
In an application embodiment, as shown in fig. 7, a database storage space guarantee method (hereinafter referred to as a data migration storage method) applied to system migration is provided, and the specific steps are as follows:
101. and setting a system migration switching plan, and setting the starting date and time of new and old system switching and the traffic of each migration through a storage space analysis processing server.
102. The migration rule base is initialized. When a new transaction is generated during the migration of the new and old systems, the incidence relation between the new and old data structures and the estimated data table space size are initialized, and the comparison relation of all the table structures of the new and old systems is set and the data table space is estimated.
103. The real-time space condition of the database is acquired regularly every day, and the real-time storage space condition of the database is acquired regularly every day, including the overall utilization rate of the storage space of the database and the space occupation condition (including data space, index space, fragment space and the like) of each database.
104. And checking whether the estimated value is the same as the estimated value of the current cut flow at the last time, acquiring the transaction number according to the running condition of the new system, recalculating according to the rule base and the cut flow plan data which are initialized, and judging whether the initialized parameters are accurate. If the estimated value of the storage space does not conform to the actual value, step 105 is performed to recalculate and adjust the migration rule base, otherwise step 106 is performed.
105. Recalculating and adjusting the migration rule base, and adjusting the migration rule base according to the actually calculated storage space occupation condition 104. For example, the actual occupancy of the first record of the B 'table is 5.2KB, and 5.2KB is adjusted to the table record occupancy parameter of B'.
106. And calculating the storage space requirement according to the tangential flow plan and the migration rule base, and recalculating the storage space required by the next migration according to the migration rule of the migration rule base according to the migration transaction data volume and the date set by the next tangential flow plan.
107. Judging whether the storage space utilization rate exceeds a storage space threshold value, calculating whether the storage space utilization rate exceeds the storage space utilization rate threshold value according to the storage space utilization rate of the current database and the storage space required by the next migration, if so, entering a step 108, and if not, entering a step 110. For example, if the current storage space usage rate is 70%, the next cut flow needs to occupy 12% of the storage space, and the storage space usage rate exceeds the threshold of 80%, step 108 is performed.
108. And stopping the flow switching and adjusting the flow switching plan, wherein the flow switching plan is stopped and the date and time of the flow switching plan are adjusted because the storage space utilization rate of the previous database and the storage space calculation required by the next migration exceed the storage space utilization rate threshold value.
109. And implementing data space guarantee measures, and implementing space guarantee measures according to the set data space guarantee measures. When the usage rate of the database storage space is reduced to a predetermined threshold, the process jumps to step 106, and the storage space is restored to meet the next cut flow.
110. And implementing a flow switching plan, because the storage space is enough, implementing flow switching according to the plan, adding the transaction of the test point into a test point white list, and implementing flow access to a new system.
In this embodiment, the data storage amount of the data to be stored is obtained by pre-estimation, and then the processing instruction for the target database is obtained according to the data storage amount and the storage space information of the target database, so that the target database stores the data to be stored according to the processing instruction, the data amount does not exceed the storage space of the database, and the data storage can be efficiently performed.
It should be understood that, although the steps in the flowcharts related to the embodiments as described above are sequentially displayed as indicated by arrows, the steps are not necessarily performed sequentially as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a part of the steps in the flowcharts related to the embodiments described above may include multiple steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the execution order of the steps or stages is not necessarily sequential, but may be rotated or alternated with other steps or at least a part of the steps or stages in other steps.
Based on the same inventive concept, the embodiment of the present application further provides a data storage device for implementing the above mentioned data storage method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so specific limitations in one or more embodiments of the data storage device provided below can refer to the limitations on the data storage method in the foregoing, and details are not described herein again.
In one embodiment, as shown in fig. 8, there is provided a data storage device comprising: a data storage amount obtaining module 801, a storage space information obtaining module 802, a processing instruction obtaining module 803, and a processing instruction executing module 804, wherein:
a data storage amount obtaining module 801, configured to obtain a data storage amount of data to be stored;
a storage space information obtaining module 802, which obtains storage space information of a target database for storing data to be stored;
a processing instruction obtaining module 803, configured to obtain a database processing instruction for a target database based on the data storage amount and the storage space information;
and the processing instruction executing module 804 is configured to send the database processing instruction to the target database, so that the target database stores the data to be stored according to the database processing instruction.
In one embodiment, the data storage amount obtaining module 801 is further configured to obtain the amount of data to be stored, acquiring average data storage capacity of each data to be stored; and obtaining the data storage quantity of the data to be stored based on the quantity of the data to be stored and the average data storage quantity.
In one embodiment, the data storage amount obtaining module 801 is further configured to obtain the amount of the original data; the original data are data stored in an original database, and the data to be stored are obtained by converting the original data; acquiring data conversion information for converting original data into data to be stored; and obtaining the quantity of the data to be stored based on the data conversion information and the quantity of the original data.
In one embodiment, the data storage amount obtaining module 801 is further configured to obtain the storage amount of the stored data in the target database, and obtain the amount of the stored data; the data types of the stored data and the data to be stored are the same, and the number of the stored data is multiple; and obtaining the average data storage quantity of each data to be stored based on the quantity of the stored data and the storage quantity of the stored data.
In one embodiment, the processing instruction obtaining module 803 is further configured to obtain an estimated usage rate of the target database based on the data storage amount and the storage space information; the estimated utilization rate is used for representing the utilization rate of a storage space after the target database stores data to be stored; under the condition that the estimated utilization rate is greater than or equal to a preset threshold value, setting a database processing instruction as a first instruction for increasing the storage space of a target database; and under the condition that the estimated utilization rate is smaller than a preset threshold value, setting the database processing instruction as a second instruction for storing the data to be stored. The processing instruction obtaining module 804 is further configured to send the database processing instruction to the target database when the database processing instruction is the first instruction, so that the target database performs augmentation processing on the storage space; if the estimated utilization rate is smaller than a preset threshold value after the storage space of the target database is increased, setting the database processing instruction as a second instruction; and under the condition that the database processing instruction is a second instruction, sending the database processing instruction to the target database so as to enable the target database to store the data to be stored.
In one embodiment, the processing instruction obtaining module 803 is further configured to set the database processing instruction as a third instruction for storing the data to be stored to the big data platform if the estimated utilization rate is greater than or equal to the preset threshold value after the storage space of the target database is increased; and under the condition that the database processing instruction is a third instruction, sending the database processing instruction to the target database so that the target database stores the database processing instruction as the data to be stored in the big data platform.
The various modules in the data storage device described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent of a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 9. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory being in a non-volatile storage medium the execution of the operating system and the computer program provides an environment. The database of the computer device is used for storing data storage data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program when executed by a processor implements a data storage method.
It will be appreciated by those skilled in the art that the configuration shown in fig. 9 is a block diagram of only a portion of the configuration associated with the present application, and is not intended to limit the computing device to which the present application may be applied, and that a particular computing device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is further provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method embodiments when executing the computer program.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
In an embodiment, a computer program product is provided, comprising a computer program which, when being executed by a processor, carries out the steps of the above-mentioned method embodiments.
It should be noted that, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or sufficiently authorized by each party.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include a Read-Only Memory (ROM), a magnetic tape, a floppy disk, a flash Memory, an optical Memory, a high-density embedded nonvolatile Memory, a resistive Random Access Memory (ReRAM), a Magnetic Random Access Memory (MRAM), a Ferroelectric Random Access Memory (FRAM), a Phase Change Memory (PCM), a graphene Memory, and the like. Volatile Memory can include Random Access Memory (RAM), external cache Memory, and the like. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), among others. The databases referred to in various embodiments provided herein may include at least one of relational and non-relational databases. The non-relational database may include, but is not limited to, a block chain based distributed database, and the like. The processors referred to in the various embodiments provided herein may be, without limitation, general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, or the like.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, and these are all within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.
Claims (10)
1. A method of data storage, the method comprising:
acquiring the data storage capacity of data to be stored;
acquiring storage space information of a target database for storing the data to be stored;
obtaining a database processing instruction aiming at the target database based on the data storage amount and the storage space information;
and sending the database processing instruction to the target database so that the target database stores the data to be stored according to the database processing instruction.
2. The method according to claim 1, wherein the number of the data to be stored is plural; the method for acquiring the data storage capacity of the data to be stored comprises the following steps:
acquiring the quantity of the data to be stored and the average data storage quantity of each data to be stored;
and obtaining the data storage quantity of the data to be stored based on the quantity of the data to be stored and the average data storage quantity.
3. The method according to claim 2, wherein the obtaining the amount of the data to be stored comprises:
acquiring the quantity of original data; the original data is data stored in an original database, and the data to be stored is obtained by converting the original data;
acquiring data conversion information for converting the original data into the data to be stored;
and obtaining the quantity of the data to be stored based on the data conversion information and the quantity of the original data.
4. The method according to claim 2, wherein the obtaining of the average data storage amount of each data to be stored comprises:
obtaining the storage amount of the stored data in the target database and obtaining the quantity of the stored data; the data category of the stored data is the same as that of the data to be stored, and the number of the stored data is multiple;
and obtaining the average data storage quantity of each data to be stored based on the quantity of the stored data and the storage quantity of the stored data.
5. The method of claim 1, wherein obtaining database processing instructions for the target database based on the data storage amount and the storage space information comprises:
obtaining the estimated utilization rate of the target database based on the data storage capacity and the storage space information; the estimated utilization rate is used for representing the utilization rate of a storage space of the target database after the data to be stored is stored;
setting the database processing instruction as a first instruction for increasing the storage space of the target database under the condition that the estimated utilization rate is greater than or equal to a preset threshold value;
under the condition that the estimated utilization rate is smaller than the preset threshold value, setting the database processing instruction as a second instruction for storing the data to be stored;
sending the database processing instruction to the target database to enable the target database to store the data to be stored according to the database processing instruction, wherein the method comprises the following steps:
under the condition that the database processing instruction is a first instruction, sending the database processing instruction to the target database so that the target database performs augmentation processing on the storage space;
if the estimated utilization rate is smaller than the preset threshold value after the storage space of the target database is increased, setting the database processing instruction as the second instruction;
and under the condition that the database processing instruction is a second instruction, sending the database processing instruction to the target database so as to enable the target database to store the data to be stored.
6. The method according to claim 5, wherein, after sending the database processing instruction to the target database to cause the target database to perform the augmentation processing on the storage space if the database processing instruction is the first instruction, further comprising:
if the estimated utilization rate is larger than or equal to the preset threshold value after the storage space of the target database is increased, setting the database processing instruction as a third instruction for storing the data to be stored to a big data platform;
and under the condition that the database processing instruction is a third instruction, sending the database processing instruction to the target database, so that the target database sets the database processing instruction as the data to be stored in a big data platform.
7. A data storage device, characterized in that the device comprises:
the data storage quantity acquisition module is used for acquiring the data storage quantity of the data to be stored;
the storage space information acquisition module is used for acquiring the storage space information of a target database for storing the data to be stored;
the processing instruction acquisition module is used for acquiring a database processing instruction aiming at the target database based on the data storage amount and the storage space information;
and the processing instruction execution module is used for sending the database processing instruction to the target database so as to enable the target database to store the data to be stored according to the database processing instruction.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 6.
9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program realizes the steps of the method of any one of claims 1 to 6 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210982516.5A CN115269558A (en) | 2022-08-16 | 2022-08-16 | Data storage method, apparatus, device, storage medium and program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210982516.5A CN115269558A (en) | 2022-08-16 | 2022-08-16 | Data storage method, apparatus, device, storage medium and program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115269558A true CN115269558A (en) | 2022-11-01 |
Family
ID=83752540
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210982516.5A Pending CN115269558A (en) | 2022-08-16 | 2022-08-16 | Data storage method, apparatus, device, storage medium and program product |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115269558A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117081860A (en) * | 2023-10-16 | 2023-11-17 | 金盾检测技术股份有限公司 | Distributed network security verification method and system |
-
2022
- 2022-08-16 CN CN202210982516.5A patent/CN115269558A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117081860A (en) * | 2023-10-16 | 2023-11-17 | 金盾检测技术股份有限公司 | Distributed network security verification method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11314701B2 (en) | Resharding method and system for a distributed storage system | |
| US10146469B2 (en) | Dynamic storage tiering based on predicted workloads | |
| US9733844B2 (en) | Data migration method, data migration apparatus, and storage device | |
| EP2838025B1 (en) | Storage array management method and device, and controller | |
| CN105243109A (en) | Data backup method and data processing system | |
| EP3859507A1 (en) | Method for processing stripe in storage device and storage device | |
| CN110795508B (en) | Data copying method, device, equipment and storage medium | |
| CN105573859A (en) | Data recovery method and device of database | |
| US9164849B2 (en) | Backup jobs scheduling optimization | |
| US11385828B2 (en) | Method and apparatus for calculating storage system available capacity | |
| CN111966449A (en) | Virtual machine backup management method, system, terminal and storage medium | |
| CN110597655A (en) | Fast predictive restoration method for coupling migration and erasure code-based reconstruction and implementation | |
| CN111930716A (en) | Database capacity expansion method, device and system | |
| CN115269558A (en) | Data storage method, apparatus, device, storage medium and program product | |
| CN106973091B (en) | Distributed memory data redistribution method and system, and master control server | |
| CN109634524B (en) | Data partition configuration method, device and equipment of data processing daemon | |
| CN110688065A (en) | Storage space management method, system, electronic equipment and storage medium | |
| CN113760187A (en) | Method, system, terminal and storage medium for generating deduplication IO thread based on vdbench | |
| CN109753384B (en) | Cloud host snapshot backup method and device, computer equipment and storage medium | |
| CN115756955A (en) | Data backup and data recovery method and device and computer equipment | |
| CN106933496B (en) | Manage the method and device of RAID | |
| CN113312309B (en) | Snapshot chain management method, device and storage medium | |
| CN114513498A (en) | File transmission checking method and device, computer equipment and storage medium | |
| CN113885803A (en) | Data storage method and device, electronic equipment and storage medium | |
| CN117435135B (en) | Method, device and system for recovering storage space related to repeated data deletion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |