CN113495883A - Data storage method and device for database - Google Patents
Data storage method and device for database Download PDFInfo
- Publication number
- CN113495883A CN113495883A CN202010203414.XA CN202010203414A CN113495883A CN 113495883 A CN113495883 A CN 113495883A CN 202010203414 A CN202010203414 A CN 202010203414A CN 113495883 A CN113495883 A CN 113495883A
- Authority
- CN
- China
- Prior art keywords
- memory
- data
- database
- storage
- log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/219—Managing data history or versioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Abstract
A data storage method and device for a database are used for providing a data storage mode of the database which is efficient and low in maintenance cost. In the application, the data processing device can record a first log when detecting the data change of the database, the first log is preferentially stored in the first memory, when the data processing device determines that the storage space in the first memory is insufficient, for example, less than a first threshold value, the data processing device can switch to the second memory to continue recording the first log, and the first memory and the second memory are arranged in different devices. The logs of the database are not limited in the local storage, when the storage space in the first storage is insufficient, the logs can be switched to other storages to be stored continuously, so that the data storage mode of the database is more efficient, the plurality of storages can be deployed in different devices, a large local disk is not needed, and the disk maintenance cost of the database can be effectively reduced.
Description
Technical Field
The present application relates to the field of communications technologies, and in particular, to a data storage method and apparatus for a database.
Background
At present, files related to a database, such as logs recording data changes in the database, and sorting files generated by executing external discharge, are usually placed in the same disk, such as local disks or cloud disks.
For the situation that the files related to the database service are placed in the local disk, more local disk space is needed, and the maintenance cost is increased.
For the situation that the file related to the database service is placed on the cloud disk, reading the data in the cloud disk needs to be performed by means of Remote Direct Memory Access (RDMA) technology, and the data reading and writing efficiency is poor.
In summary, a data storage method for a database with high efficiency and low maintenance cost is needed.
Disclosure of Invention
The application provides a data storage method and device for a database, which are used for providing a data storage mode of the database with high efficiency and low maintenance cost.
In a first aspect, an embodiment of the present application provides a data storage method for a database, where the method is executed by a data processing apparatus, and in the method: the data processing apparatus is capable of recording a first log when detecting a data change of the database, the first log being preferentially stored in a first memory, and when the data processing apparatus determines that a storage space in the first memory is insufficient, for example, less than a first threshold, it may switch to a second memory to continue recording the first log, the first memory and the second memory being disposed in different devices.
By the method, the logs of the database are not limited in the local storage, when the storage space in the first storage is insufficient, the logs can be switched to other storages to be stored continuously, so that the data storage mode of the database is more efficient, the plurality of storages can be deployed in different devices, a large local disk is not needed, and the disk maintenance cost of the database can be effectively reduced.
In one possible implementation, in addition to the database log, the sort file generated by the database during the sorting process may also be stored in a similar manner. For example, when the data processing apparatus receives a database sorting request, the data in the database may be sorted according to the received database sorting request; the data processing device may further generate a sorting file during sorting of the data, where the sorting file may be stored in the first memory, and when the data processing device determines that the storage space of the first memory is smaller than the second threshold, the data processing device may switch to the second memory to continue storing the second memory. The above manner is only described by taking the received database sorting request as an example, in some scenarios, if the data in the database needs to be queried, the data processing device will also sort the data in the database to generate a sort file, and the sort file may also store the sort file in the above manner.
By the method, the sorting files generated in the data sorting process in the database can be stored in the plurality of memories, so that the data storage efficiency of the database is improved, the plurality of memories are distributed and deployed, and the maintenance cost of the local disk is reduced.
In a possible implementation manner, for the database, an extended memory and a memory may also be provided, where the extended memory and the memory can be used to store data with a high read-write frequency in the database, and the extended memory and the memory may be deployed in the first storage or may be other storages independent from the first storage. The data processing device can store the data with the reading and writing frequency larger than a third threshold value in the data of the database in the extended memory based on the reading and writing frequency of the data in the database; when the data processing device needs to read the data in the extended memory, the data can be read from the extended memory to the memory, and the data processing device reads the data from the memory.
By the method, the storage space of the memory can be effectively expanded by the arrangement of the expanded memory, so that more data can be stored in the expanded memory, and the data can be ensured to be efficiently read and written.
In a possible implementation manner, the data processing device can also eliminate data with lower read-write frequency in the extended memory in time, and the data processing device detects the read-write frequency of the data in the extended memory firstly; if the data with the read-write frequency lower than the third threshold value is stored in the extended memory, the data with the read-write frequency lower than the third threshold value in the extended memory can be eliminated.
By the method, the data processing device can eliminate cold data in the extended memory in time and effectively utilize the storage space in the extended memory.
In a possible implementation, the data processing apparatus may further back up a first log stored in the first memory and the second memory, and may clear the first log after the backup of the first log is completed; if there is a data change in the database, the data processing apparatus may continue to record the second log of the database in the first memory according to the data change of the database.
By the method, after the first memory has free memory space (such as clearing the first journal), the journal can be continuously stored in the first memory, so that the memory space of the first memory can be effectively utilized.
In a possible implementation manner, the data processing apparatus may delete the sorting file after finishing sorting the data in the database according to the sorting file.
By the method, the data processing device deletes the sequencing file in time, and can ensure that the first memory and the second memory have free storage space so as to store other effective data.
In a possible implementation manner, the data processing apparatus may further record a corresponding relationship between the data page in the extended memory and the data page identifier, for example, the corresponding relationship is stored in the nonvolatile memory, and after the device where the first storage is located is restarted, the data processing apparatus may reassemble the extended memory in the first storage according to the corresponding relationship.
By the method, the data processing device can quickly organize the extended memory after the equipment is restarted.
In a second aspect, an embodiment of the present application further provides a data processing apparatus, and for beneficial effects, reference may be made to the description of the first aspect, which is not described herein again. The apparatus has the functionality to implement the actions in the method instance of the first aspect described above. The functions can be realized by hardware, and the functions can also be realized by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above-described functions. In a possible design, the structure of the apparatus includes a processing unit, a determining unit, and a switching unit, and these units may perform corresponding functions in the method example of the first aspect, for which specific reference is made to the detailed description in the method example, and details are not repeated here.
In a third aspect, an embodiment of the present application further provides a computing device, where the computing device includes a processor and a memory, and may further include a communication interface, where the processor executes program instructions in the memory to perform the method provided in the first aspect or any possible implementation manner of the first aspect, and the memory is coupled to the processor and stores program instructions and data necessary for performing data synchronization. The communication interface is used for communicating with other devices (such as client devices).
In a fourth aspect, the present application provides a computing device system comprising at least one computing device. Each computing device includes a memory and a processor. A processor of at least one computing device may be configured to access the code in the memory to perform the method provided by the first aspect or any one of the possible implementations of the first aspect.
In a fifth aspect, the present application provides a non-transitory readable storage medium which, when executed by a computing device, performs the foregoing first aspect or any possible implementation manner of the first aspect. The storage medium stores a program therein. The storage medium includes, but is not limited to, volatile memory such as random access memory, and non-volatile memory such as flash memory, Hard Disk Drive (HDD), and Solid State Drive (SSD).
In a sixth aspect, the present application provides a computing device program product comprising computer instructions that, when executed by a computing device, may perform the first aspect or any possible implementation of the first aspect. The computer program product may be a software installation package, which may be downloaded and executed on a computing device in case it is desired to use the method as provided in the first aspect or any possible implementation manner of the first aspect.
Drawings
FIG. 1 is a block diagram of a system according to the present application;
FIG. 2 is a schematic diagram of another system configuration provided herein;
FIG. 3 is a schematic diagram of another system configuration provided herein;
FIG. 4 is a schematic diagram of a data storage method for a database provided in the present application;
FIG. 5 is a schematic structural diagram of a data processing apparatus provided in the present application;
FIG. 6 is a schematic diagram of a computing device provided by an embodiment of the present application;
fig. 7 is a schematic diagram of a computing device in a computing device system according to an embodiment of the present application.
Detailed Description
Fig. 1 is a schematic structural diagram of a system provided in the embodiment of the present application, where the system includes a data processing apparatus 100, a first memory 200, and a second memory 300.
The first memory 200 may include a volatile memory (volatile memory), such as a Random Access Memory (RAM). The first memory may further include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a Hard Disk Drive (HDD) or a solid-state drive (SSD). The first memory may also comprise a combination of the above categories. The first memory 200 is similar to the second memory 300, and reference may be made to the foregoing specifically, which is not described herein again. The first storage 200 and the second storage 300 are deployed in different data centers and located in different devices, for example, the first storage 200 is a local disk, and the second storage 300 is a cloud disk.
The first memory 200 and the second memory 300 are used for storing data in a database, and related files of the database, such as logs (e.g., physical logs and logical logs), sorting files, and the like.
The first storage 200 may further include an extended memory and a memory, where the extended memory and the memory are used to store data with high read/write frequency in the database.
The data processing device 100 may be a hardware device, such as: a server, a terminal computing device, etc., or a software device, specifically a set of software systems running on a hardware computing device. The position where the data processing apparatus 100 is disposed is not limited in the embodiment of the present application. For example, as shown in fig. 2, the data processing apparatus 100 may operate in a cloud computing device system (including at least one cloud computing device, for example, a server, and the like, and in this embodiment, the cloud computing device system is a data center), may also operate in an edge computing device system (including at least one edge computing device, for example, a server, a desktop, and the like, and in this embodiment, the edge computing device system is a data center), and may also operate in various terminal computing devices, for example: notebook computers, personal desktop computers, and the like.
The data processing apparatus 100 may also be logically configured by a plurality of parts, for example, the data processing apparatus 100 may include a processing unit, a determining unit and a switching unit, and each component in the data processing apparatus 100 may be respectively deployed in different systems or servers. For example, as shown in fig. 3, each part of the apparatus may operate in three environments, namely, a cloud computing device system, an edge computing device system, or a terminal computing device, respectively, or may operate in any two of the three environments. The cloud computing device system, the edge computing device system and the terminal computing device are connected through communication paths, and can communicate with each other and transmit data. The data storage method for the database provided by the embodiment of the application is cooperatively executed by each combined part of the data processing device 100 operating in three environments (or any two of the three environments).
The first memory 200 may be deployed on the same system or in the same hardware device as the data processing apparatus 100, so that the data processing apparatus 100 can more efficiently read data from the first memory 200, and the deployment location of the second memory 300 is not limited in this application, and it is only required to ensure that the first memory 200 and the second memory 300 are deployed in different data centers.
In the embodiment of the present application, the data processing apparatus 100 can generate a log according to the change status of data in the database, preferentially store the generated log in the first memory 200 when the free storage space in the first memory 200 is greater than the first threshold, gradually decrease the storage space in the first memory 200 as the log is generated and stored, and switch the data processing apparatus 100 to the second memory 300 to continue to store the log when the free storage space in the first disk is less than the first threshold. Therefore, the logs can be stored in at least two different first memories 200, so that the utilization rates of different memories can be effectively improved, and the logs of the database can be completely stored.
The data storage method for the database provided by the embodiment of the application is provided below with reference to fig. 4, and the method includes:
step 401: the data processing apparatus 100 detects a data change of the database. Data changes to the database include, but are not limited to: insertion (insert), deletion (delete), update (update) of data. The data insertion refers to adding new data into the database, the data deletion refers to deleting data in the database, and the data update refers to changing one data in the database into another data.
Step 402: the data processing apparatus 100 generates a first log according to a data change of the database.
In the database, the log is used for recording data changes sent by the database of the meter, such as data insertion, deletion, update and the like. When data in the database is changed, the data processing apparatus 100 generates a corresponding log after checking the data change, and in the embodiment of the present application, the log generated in step 402 is distinguished from the log generated by the subsequent data processing apparatus 100, and is referred to as a first log.
Step 403: in the case where the free memory space in the first memory 200 is not less than the first threshold value, the data processing apparatus 100 stores the first log in the first memory 200.
Step 404: the data processing apparatus 100 switches to the second memory 300 to continue storing the first log when determining that the free storage space in the first memory 200 is less than the first threshold.
In the embodiment of the present application, the storage order of the logs is first storage 200- > second storage 300. That is, the data processing apparatus 100 preferentially stores the first log in the first memory 200, and switches to the second memory 300 to continue storing the first log when the free storage space in the first memory 200 is insufficient (e.g., smaller than the first threshold).
In the database, a fixed file is configured to store the log, and the file may be referred to as a log file, the size of the log file is fixed and usually a preset value, and when the size of the first log generated by the data processing apparatus 100 reaches the preset value along with the data change of the database, the database cannot continue the data change. At this time, the first log needs to be backed up, and the first log can be erased after the backup is completed, that is, the first log stored in the first memory 200 and the second memory 300 is deleted.
After the first log is cleared, if there is still a data change in the database, the data processing apparatus 100 may continue to generate the log, and for convenience of description, the log generated by the data processing apparatus 100 is referred to as a second log, and the data processing apparatus 100 preferentially stores the second log in the first memory 200, and switches to the second memory 300 to continue to store the second memory 300 when the free storage space of the first memory 200 is insufficient, similarly to the first log.
While the storage manner of the logs in the first memory 200 and the second memory 300 is described in the foregoing, the data processing apparatus 100 may also use a similar storage manner for other data (such as the sorting file) of the database.
When receiving the data sorting request, the data processing apparatus 100 may sort the data in the database according to the received request to feed back the sorted data.
When the data processing apparatus 100 sorts the data in the database, a sort file is generated, and the sort file records the sequence of the sorted data. The sorting file can also record the storage position of the data, the index number of the data and the like.
After the data processing apparatus 100 generates the ordered file, the ordered file may be preferentially stored in the first memory 200, the free storage space in the first memory 200 is less and less as the ordered file is stored in the first memory 200, and when the free storage space in the first memory 200 is insufficient (e.g., smaller than the second threshold), the data processing apparatus switches to the second memory 300 to continue storing the ordered file.
When the data processing apparatus 100 finishes sorting the data in the database and determines a sorting result from the sorting file, the sorting file may be deleted.
That is, the sorted file is stored in the order of the first memory 200- > the second memory 300, preferentially stored in the first memory 200, and secondly stored in the second memory 300.
In the above description, the request received by the data processing apparatus 100 is taken as a data sorting request as an example, when the data processing apparatus 100 has received a data query request, in order to quickly query the data processing apparatus 100 and also sort the data in the database, a sorting file is generated, and a storage manner of the sorting file is similar to the foregoing manner, and is not described here again.
As can be seen from the above, the data processing apparatus 100 can store the log and the sort file in the first memory 200 and the second memory 300, and the data storage is more flexible and the data storage space is effectively expanded by adopting such a data storage method.
As a possible implementation manner, the first storage 200 may be a local disk of a database, and the second storage 300 is deployed in a cloud computing device system, that is, the second storage 300 is a cloud disk, related files or data of the database may be expanded to a cloud storage, and files related to the database may be distributed in the local disk and the cloud disk, which has both the extensibility of the cloud disk and can effectively save the maintenance cost.
It should be noted that, in the embodiment of the present application, only two memories are included in the system as an example, in some scenarios, more memories may be included, the position of data storage is more flexible, and the utilization rate of the memories may also be effectively improved.
In addition to the aforementioned data storage manner, in the embodiment of the present application, a memory (buffer pool) in a device in which the database is deployed may be extended, for example, an extended buffer pool (extended buffer pool) is set in the first storage.
To further highlight the role of the extended memory, the memory is first described, the read/write frequencies of the data in the database are usually different, and when the data in the database is managed by the conventional data processing apparatus 100, the data with higher read/write frequency in the database may be placed in the memory. Generally, the data read-write frequency in the memory is high, and the data processing device 100 can preferentially acquire data from the memory, so that the read-write data of the data can be effectively improved, and the data processing efficiency of the data processing device 100 is further improved.
However, the space of the memory is limited, and all data with high read/write frequency cannot be stored in the memory, and the data stored in the memory may also be updated, so that some data with high read/write frequency are eliminated, and the data processing apparatus 100 needs to load the data from the second storage 300, and therefore, an extended memory may be added.
The extended memory can store data with a data read-write frequency greater than a third threshold value, so that the extended memory stores data with a data read-write frequency greater than the third threshold value. The data processing apparatus 100 may periodically update the data in the extended memory, store the data with the data read-write frequency greater than the third threshold in the extended memory, and may also eliminate the data with the read-write frequency lower than the third threshold stored in the extended memory.
The following describes the manner of data migration into the extended memory and data elimination in the extended memory respectively:
(1) and migrating the data into the extended memory.
When migrating data into the extended memory, the data processing apparatus 100 may first locate the data to be migrated. The data processing apparatus 100 may use the data in the memory as the data to be migrated.
Data is typically stored in memory, and data in a database, such as a page of data (page), is typically organized in units of data of a fixed size. The data stored in each data page (page) is the same size, and each data page is configured with an Identification (ID). The data processing apparatus 100 can realize fast positioning of the data page through identification of each data page in the memory.
As a possible implementation, the data pages and the identifications of the data pages may be organized in the first memory 200 and the second memory 300 by using a hash table (hash map), and the data processing apparatus 100 may locate the data pages according to the identifications of the data pages based on the hash table.
(2) And migrating the data out of the extended memory.
When the data processing apparatus 100 eliminates data from the extended memory, the data to be migrated may be located first. For example, the data processing apparatus 100 may determine to expand the memory to find the least frequently used data page by using a page replacement algorithm, such as Least Recently Used (LRU), where the data in the data page is the data to be migrated.
When the data processing apparatus 100 needs to read data in the database, the data may be preferentially obtained from the memory or the extended memory, and the extended memory further extends the data amount of the data that can be stored in the memory, so as to ensure that more data with higher read-write frequency can be stored in the first storage, and improve the data read-write efficiency.
As a possible implementation manner, the data processing apparatus 100 may further record a corresponding relationship between the data page in the extended memory and the identifier of the data page, store the corresponding relationship in the nonvolatile memory, and when the first device is restarted, may directly call the previously stored corresponding relationship to reorganize the extended memory, so that the data page stored in the extended memory may be quickly located.
Based on the same inventive concept as the method embodiment, the embodiment of the present application further provides a data processing apparatus, which is configured to execute the method executed by the data processing apparatus 100 in the method embodiment shown in fig. 4. As shown in fig. 5, the data processing apparatus 500 includes a processing unit 501, a determining unit 502, and a switching unit 503, which may be software modules. Specifically, in the data processing apparatus 500, the units are connected to each other through a communication path.
The processing unit 501 is configured to record a first log of the database in the first memory according to a data change of the database. The processing unit 501 may perform steps 401-403 as shown in FIG. 4.
A determining unit 502 is configured to determine that a storage space of the first memory is smaller than a first threshold. The determining unit 502 may perform a method of determining that the storage space of the first memory is less than the first threshold in step 404 as shown in fig. 4.
A switching unit 503, configured to, after the determining unit 502 determines that the storage space of the first memory is smaller than the first threshold, cause the processing unit 501 to switch to a second memory to record the first log of the database, where the first memory and the second memory are disposed in different devices. The switching unit 503 may perform a method of recording the first log by switching to the second memory exactly as in step 404 shown in fig. 4.
In a possible implementation, the apparatus further comprises a sorting unit 504.
After receiving the database sorting request, the sorting unit 504 may sort the data in the database according to the received database sorting request; the processing unit 501 stores the sort file generated in the sorting process in a first memory;
the determining unit 502 may cause the processing unit 501 to switch to the second memory to store the second memory after determining that the storage space of the first memory is smaller than the second threshold.
In one possible implementation, the first storage includes an expansion memory and a memory;
the processing unit 501 may store, in the extended memory, data of which the read-write frequency is greater than the third threshold in the data of the database, and read the data in the extended memory to the memory, according to the read-write frequency of the data in the database.
In a possible implementation manner, the processing unit 501 may also eliminate data in the extended memory, whose read-write frequency is lower than the third threshold.
In a possible implementation, after the processing unit 501 switches to the first log of the second memory record database, the first log may be cleared after the backup of the first log is completed; and recording a second log of the database in the first memory according to the data change of the database.
In a possible implementation manner, after the processing unit 501 stores the second sort file generated in the sorting process in the second memory, the sort file may be deleted after the sorting unit 504 finishes sorting the data in the database according to the sort file.
In a possible implementation manner, the processing unit 501 may further record a corresponding relationship between a data page and a data identifier in the extended memory, and reassemble the extended memory according to the corresponding relationship after the device where the first storage is located is restarted.
The division of the modules in the embodiments of the present application is schematic, and only one logic function division is provided, and in actual implementation, there may be another division manner, and in addition, each functional module in each embodiment of the present application may be integrated in one processor, may also exist alone physically, or may also be integrated in one module by two or more modules. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.
The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a terminal device (which may be a personal computer, a mobile phone, or a network device) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
Such as computing device 600 shown in fig. 6. The computing device 600 includes a bus 601, a processor 602, a communication interface 603, and a memory 604. The processor 602, memory 604, and communication interface 603 communicate over a bus 601.
The processor 602 may be a Central Processing Unit (CPU). The memory 604 may include a volatile memory (volatile memory), such as a Random Access Memory (RAM). The memory 604 may also include a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, an HDD, or an SSD. The memory has stored therein executable code that the processor 602 executes to perform the aforementioned data storage method (e.g., the method shown in fig. 4). The memory 604 may also include other software modules required to run processes, such as an operating system. The operating system may be LINUXTM, UNIXTM, WINDOWSTM, etc.
Specifically, the memory 604 stores the modules of the data processing apparatus 500. The memory 604 may include, in addition to the aforementioned modules, other software modules required for running processes, such as an operating system. The operating system may be LINUXTM, UNIXTM, WINDOWSTM, etc.
The present application also provides a computing device system that includes at least one computing device 700 as shown in fig. 7. The computing device 700 includes a bus 701, a processor 702, a communication interface 703, and a memory 704. The processor 702, memory 704, and communication interface 703 communicate over a bus 701. At least one computing device 700 in the system of computing devices communicates with each other via a communication path.
The processor 702 may be a CPU. The memory 704 may include volatile memory, such as random access memory. The memory 704 may also include a non-volatile memory, such as a read-only memory, flash memory, HDD, or SSD. The memory 704 has stored therein executable code that the processor 702 executes to perform any or all of the aforementioned methods of data synchronization. The memory may also include other software modules required to run processes, such as an operating system. The operating system may be LINUX, UNIX, WINDOWS, etc.
Specifically, the memory 704 stores any one or more modules of the data processing apparatus 500. The memory 704 may include, in addition to any one or more of the elements described above, other software modules required to run a process, such as an operating system. The operating system may be LINUX, UNIX, WINDOWS, etc.
At least one of the computing devices 700 in the computing device system, on each of which any one or any number of the elements of the data processing apparatus 500 are running, establishes communication with each other via a communication network. At least one computing device 700 collectively performs the aforementioned data storage operations.
The descriptions of the flows corresponding to the above-mentioned figures have respective emphasis, and for parts not described in detail in a certain flow, reference may be made to the related descriptions of other flows.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product for data synchronization comprises one or more computer program instructions for data synchronization which, when loaded and executed on a computer, cause, in whole or in part, the flow or function of data synchronization according to embodiments of the invention.
The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, from one website, computer, server, or data center over a wired (e.g., coaxial, fiber, digital subscriber line, or wireless (e.g., infrared, wireless, microwave, etc.) link to another website, computer, server, or data center. (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., DVD), or semiconductor media (e.g., SSD).
Claims (16)
1. A method of data storage for a database, the method comprising:
recording a first log of a database in the first memory according to data change of the database;
after determining that the storage space of the first storage is smaller than a first threshold value, switching to the second storage to record a first log of the database, wherein the first storage and the second storage are arranged in different devices.
2. The method of claim 1, wherein the method further comprises:
sorting the data in the database according to the received database sorting request;
storing the sorting file generated in the sorting process in the first memory;
and after the storage space of the first storage is determined to be smaller than a second threshold value, switching to the second storage to continue to store the second storage.
3. The method of claim 1 or 2, wherein the first storage includes extended memory and memory, the method further comprising:
storing data with the read-write frequency larger than a third threshold value in the data of the database in the extended memory based on the read-write frequency of the data in the database;
and reading the data in the extended memory to the memory.
4. The method of claim 3, wherein the method further comprises:
and eliminating the data with the reading and writing frequency lower than a third threshold value in the extended memory.
5. The method of any of claims 1 to 4, wherein after the switching to the second storage to record the first log of the database, further comprising:
clearing the first log after the backup of the first log is completed;
and recording a second log of the database in the first memory according to the data change of the database.
6. The method according to any one of claims 1 to 5, wherein after storing the second sorted file generated in the sorting process in the second memory, the method further comprises:
and deleting the sorting file after finishing sorting the data in the database according to the sorting file.
7. The method of claim 3 or 4, further comprising:
recording the corresponding relation between the data page and the data identification in the extended memory;
and after the equipment where the first storage is located is restarted, the extended memory is recombined according to the corresponding relation.
8. A data processing apparatus, characterized in that the apparatus comprises a processing unit, a determining unit, and a switching unit:
the processing unit is used for recording a first log of the database in the first memory according to the data change of the database;
the determining unit is used for determining that the storage space of the first memory is smaller than a first threshold value;
the switching unit is configured to, after the determining unit determines that the storage space of the first memory is smaller than the first threshold, switch the processing unit to the second memory to record the first log of the database, where the first memory and the second memory are deployed in different devices.
9. The apparatus of claim 8, wherein the apparatus further comprises a sorting unit to:
the sorting unit is used for sorting the data in the database according to the received database sorting request;
the processing unit is used for storing the sorting file generated in the sorting process in the first memory;
the determining unit is further configured to determine that the storage space of the first memory is smaller than a second threshold;
the switching unit is further configured to switch the processing unit to the second memory to store the second memory after the determining unit determines that the storage space of the first memory is smaller than a second threshold.
10. The apparatus of claim 8 or 9, wherein the first storage includes an expansion memory and a memory;
the processing unit is further configured to store, in the extended memory, data of which the read-write frequency is greater than a third threshold value in the data of the database based on the read-write frequency of the data in the database, and read the data in the extended memory to the memory.
11. The apparatus of claim 10,
and the processing unit is also used for eliminating the data with the reading and writing frequency lower than a third threshold value in the extended memory.
12. The apparatus according to any of claims 8 to 11, wherein the processing unit, after switching to the second memory to record the first log of the database, is further configured to:
clearing the first log after the backup of the first log is completed;
and recording a second log of the database in the first memory according to the data change of the database.
13. The apparatus according to any one of claims 9 to 12, wherein after the processing unit stores a second sort file generated in the sorting process in the second memory, the processing unit is further configured to:
and deleting the sorting file after finishing sorting the data in the database according to the sorting file.
14. The apparatus according to claim 10 or 11, wherein the processing unit is further configured to record a correspondence between a data page in the extended memory and a data identifier;
and after the equipment where the first storage is located is restarted, the extended memory is recombined according to the corresponding relation.
15. A computing device, wherein the computing device comprises a processor and a memory;
the memory to store computer program instructions;
the processor invokes computer program instructions in the memory to perform the method of any of claims 1-7.
16. A computer-readable storage medium having stored thereon computer-executable instructions for causing a computer to perform the method of any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010203414.XA CN113495883A (en) | 2020-03-20 | 2020-03-20 | Data storage method and device for database |
| PCT/CN2021/075501 WO2021184996A1 (en) | 2020-03-20 | 2021-02-05 | Data storage method and apparatus for database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010203414.XA CN113495883A (en) | 2020-03-20 | 2020-03-20 | Data storage method and device for database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113495883A true CN113495883A (en) | 2021-10-12 |
Family
ID=77769685
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010203414.XA Pending CN113495883A (en) | 2020-03-20 | 2020-03-20 | Data storage method and device for database |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN113495883A (en) |
| WO (1) | WO2021184996A1 (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115391098A (en) * | 2022-08-25 | 2022-11-25 | 北京有竹居网络技术有限公司 | Big data analysis method and device, edge node and cloud server |
| CN115658328B (en) * | 2022-12-07 | 2023-10-03 | 摩尔线程智能科技(北京)有限责任公司 | Device and method for managing storage space, computing device and chip |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080148270A1 (en) * | 2006-12-15 | 2008-06-19 | International Business Machines Corporation | Method and implementation for storage provisioning planning |
| US9465899B2 (en) * | 2013-03-15 | 2016-10-11 | Freescale Semiconductor, Inc. | Method for provisioning decoupling capacitance in an integrated circuit |
| CN103838521B (en) * | 2014-02-28 | 2017-02-08 | 华为技术有限公司 | Data processing method and data processing device |
| CN104539669B (en) * | 2014-12-17 | 2018-02-13 | 中国电子科技集团公司第十五研究所 | A kind of method of data synchronization based on mobile terminal |
| CN110413684B (en) * | 2018-04-25 | 2023-05-02 | 杭州海康存储科技有限公司 | Database synchronization method, device and system |
-
2020
- 2020-03-20 CN CN202010203414.XA patent/CN113495883A/en active Pending
-
2021
- 2021-02-05 WO PCT/CN2021/075501 patent/WO2021184996A1/en active Application Filing
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021184996A1 (en) | 2021-09-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11288235B2 (en) | Synchronized data deduplication | |
| US11327799B2 (en) | Dynamic allocation of worker nodes for distributed replication | |
| US20200348852A1 (en) | Distributed object replication architecture | |
| US11010240B2 (en) | Tracking status and restarting distributed replication | |
| US11349915B2 (en) | Distributed replication and deduplication of an object from a source site to a destination site | |
| US9336224B2 (en) | Systems and methods for providing increased scalability in deduplication storage systems | |
| US10839016B2 (en) | Storing metadata in a cuckoo tree | |
| CN108733306B (en) | File merging method and device | |
| CN105095300A (en) | Method and system for database backup | |
| US9015519B2 (en) | Method and system for cluster wide adaptive I/O scheduling by a multipathing driver | |
| US11093442B1 (en) | Non-disruptive and efficient migration of data across cloud providers | |
| US9778927B2 (en) | Storage control device to control storage devices of a first type and a second type | |
| CN110998537A (en) | Expired backup processing method and backup server | |
| CN113495883A (en) | Data storage method and device for database | |
| CN104750432A (en) | Data storage method and device | |
| CN110618789B (en) | Method and device for deleting repeated data | |
| US11392546B1 (en) | Method to use previously-occupied inodes and associated data structures to improve file creation performance | |
| CN115390754A (en) | Hard disk management method and device | |
| CN113238891A (en) | Backup deletion method and system based on backup chain | |
| US10795596B1 (en) | Delayed deduplication using precalculated hashes | |
| CN108733753B (en) | File reading method and application entity | |
| CN117806528A (en) | Data storage method and device | |
| CN117743024A (en) | Restoring backup from remote storage | |
| CN115933981A (en) | Data migration method, device, distributed system and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220214 Address after: 550025 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an New District, Guiyang City, Guizhou Province Applicant after: Huawei Cloud Computing Technology Co.,Ltd. Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Applicant before: HUAWEI TECHNOLOGIES Co.,Ltd. |
|
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |