CN113296683B - Data storage method, device, server and storage medium - Google Patents

Abstract

The embodiment of the invention provides a data storage method, a data storage device, a server and a storage medium, wherein the method comprises the following steps: writing first data into the memory; if the written first data in the memory reaches the preset data volume, first record information is generated, wherein the first record information comprises first identification information corresponding to the first data with the preset data volume in the memory and second identification information corresponding to second data stored in the database. Copying the first data with the preset data volume to a database to generate second recorded information, wherein the second recorded information comprises second identification information and third identification information corresponding to the first data with the preset data volume, and the third identification information is identification information corresponding to the first data with the preset data volume in the database. And processing data read-write transaction according to the first recording information and the second recording information. By adopting the scheme, the normal operation of data reading and writing transactions can be ensured not to be influenced in the data storage process.

Description

Data storage method, device, server and storage medium

technical field

The present invention relates to the technical field of data processing, and in particular, to a data storage method, device, server and storage medium.

Background technique

Data can be written to memory before writing data to disk. When the data in the memory accumulates to a certain amount, it will trigger the transfer of the data in the memory to the disk. This operation can be called data transfer (DeltaMerge).

During the Delta Merge process, the transferred data will temporarily reside in storage space that is neither in memory nor on disk. At this time, if a data read/write transaction is received, the transferred data cannot be queried, so the process of Delta Merge will affect the normal operation of the data read/write transaction.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a data storage method, apparatus, device, and storage medium, so as to ensure that the process of storing data in the database does not affect the normal operation of data read and write transactions.

In a first aspect, an embodiment of the present invention provides a data storage method, the method comprising:

Write first data into the memory, the first data is the data to be stored in the database;

If the first data written in the memory reaches a preset amount of data, first record information is generated, and the first record information includes first identification information and second identification information, and the first identification information is The identification information corresponding to the first data of the preset data volume in the memory, and the second identification information is the identification information corresponding to the second data stored in the database in the database;

copying the first data of the preset data amount to the database;

generating second record information, the second record information includes the second identification information and third identification information corresponding to the first data of the preset data volume, and the third identification information is the preset Identification information corresponding to the first data of the data volume in the database;

Data read and write transactions are processed according to the first record information and the second record information.

In a second aspect, an embodiment of the present invention provides a data storage device, the device comprising:

a writing module, configured to write first data into the memory, where the first data is the data to be stored in the database;

A generating module is used to generate first record information when the first data written in the memory reaches a preset data amount, and the first record information includes first identification information and second identification information, and the The first identification information is the identification information corresponding to the first data of the preset data amount in the memory, and the second identification information is the identification corresponding to the second data stored in the database in the database information;

a copying module, configured to copy the first data of the preset data amount to the database;

The generating module is configured to generate second record information, the second record information includes the second identification information and the third identification information corresponding to the first data of the preset data volume, the third identification information The identification information is the identification information corresponding to the first data of the preset data volume in the database;

The processing module is configured to process data read and write transactions according to the first record information and the second record information.

In a third aspect, an embodiment of the present invention provides a server, including: a memory and a processor; wherein, executable code is stored on the memory, and when the executable code is executed by the processor, the processing is executed. The controller executes the data storage method described in the first aspect of the embodiment of the present invention.

In a fourth aspect, an embodiment of the present invention provides a non-transitory machine-readable storage medium, where executable code is stored on the non-transitory machine-readable storage medium, and when the executable code is executed by a processor of a server , causing the processor to execute the data storage method described in the first aspect of the embodiment of the present invention.

With the method provided by the embodiment of the present invention, in the process of storing the data in the memory into the database, when the data in the memory reaches the preset data amount, the preset data amount is taken as the unit, and the memory is copied by means of copying. The data with the preset amount of data is copied to the database, so that one piece of data is still kept in the memory, and only the other piece of data is transferred, so that the currently transferred data can still be queried in the memory, avoiding the need for data storage. The problem that the transferred data cannot be queried during the transfer process. In the method provided by the embodiment of the present invention, the data identification information (reflecting the storage location of the data) stored in the memory and the database during the data transfer process and after the data transfer is completed can also be recorded by the first record information and the second record information. , when a data read/write transaction is triggered, the data read/write transaction may be executed based on the trigger time of the data read/write transaction using the corresponding record information, so as to complete the transaction submission.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

1 is a flowchart of a data storage method according to an embodiment of the present invention;

2 is a schematic diagram of generating record information according to an embodiment of the present invention;

FIG. 3 is a flowchart of a data replication method provided by an embodiment of the present invention;

4 is a schematic diagram of a storage format conversion provided by an embodiment of the present invention;

5 is a schematic diagram of a data storage method provided by an embodiment of the present invention;

6 is a schematic structural diagram of a data storage device according to an embodiment of the present invention;

FIG. 7 is a schematic structural diagram of a server according to an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a," "the," and "the" as used in the embodiments of the present invention and the appended claims are intended to include the plural forms as well, unless the context clearly dictates otherwise, "a plurality" Generally at least two are included.

Depending on the context, the words "if", "if" as used herein may be interpreted as "at" or "when" or "in response to determining" or "in response to detecting". Similarly, the phrases "if determined" or "if detected (the stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detected (the stated condition or event)," depending on the context )" or "in response to detection (a stated condition or event)".

In addition, the sequence of steps in the following method embodiments is only an example, and is not strictly limited.

Let's first explain a concept involved in this article:

A data read/write transaction, also referred to as a transaction, is the smallest logical unit of work for accessing a database in order to achieve a specific service function, and consists of a sequence of operations. Only when all operations contained in this sequence of operations complete successfully can the database transition from one state to another. If an error occurs in any operation in this sequence of operations, the previously completed operation needs to be rolled back. That is, all operations in the same transaction are either executed correctly or not executed at all.

The transaction processing method provided by the embodiment of the present invention may be executed by a server, which serves as a hardware carrier of the database. Specifically, an application program may be deployed in the server, and a certain process may be started to execute the transaction processing method.

The data storage method provided by the embodiment of the present invention can be applied to a scenario in which data is stored in a database. When data needs to be stored in the database, the data is not directly stored in the database, but is first written to the memory. Whenever the data written in the memory reaches a certain amount, a certain amount of data will be triggered. Data transfer operations from in-memory to database. In the related art, if a data transfer operation is triggered and a data read/write transaction is received during the data transfer process, the transferred data cannot be read, which will result in data loss, and the data read cannot be successfully executed. Write transactions, in general, the process of data storage will interfere with the execution process of data read and write transactions. The data storage method provided by the embodiment of the present invention can avoid this problem, so that the data storage process does not interfere with the execution process of the data read-write transaction.

The execution process of the data storage method provided herein is described below with reference to the following embodiments.

FIG. 1 is a flowchart of a data storage method provided by an embodiment of the present invention. As shown in FIG. 1 , the method includes the following steps:

101. Write first data into a memory, where the first data is data to be stored in a database.

102. If the first data written in the memory reaches the preset data amount, then generate first record information, the first record information includes first identification information and second identification information, and the first identification information is the preset data amount. The identification information corresponding to the first data in the memory, and the second identification information is the identification information corresponding to the second data stored in the database in the database.

103. Copy the first data of the preset data amount to the database.

104. Generate second record information, the second record information includes second identification information and third identification information corresponding to the first data of the preset data amount, and the third identification information is the first identification information of the preset data amount. The identification information corresponding to the data in the database.

105. Perform data read and write transactions according to the first record information and the second record information.

First summarize the core idea of the data recommendation solution provided by this embodiment: in the process of storing the first data written in the memory to the database, when the first data in the memory reaches the preset amount of data, the The first data of the data volume is copied. For convenience of description, hereinafter, Memtable is used to represent the first data of the preset data volume in the memory, and Memtable' is used to represent the data obtained by copying the first data of the preset data volume in the memory. After getting the Memtable', the Memtable' can be moved from memory to the database. At the same time, the first record information and the second record information can respectively record the identification information of the data stored in the memory and the database during the data transfer process and after the data transfer is completed (reflecting the storage location of the data), so that when a certain data is read When the write transaction is triggered, the data can be read and written by using the corresponding record information based on the trigger time of the data read and write transaction.

In practical applications, the above-mentioned preset data amount can be set according to actual requirements, for example, it can be 500MB or the like. Reasonable setting of the preset data amount can improve the efficiency of data storage and avoid long-term interference to data read and write transactions.

The above-mentioned identification information in this embodiment may be, but is not limited to, a storage location or a location index, etc., and corresponding data can be located through the identification information. Taking the above Memtable as an example, the corresponding first identification information may be a storage location of the Memtable in the memory.

In this embodiment of the present invention, only one Memtable may be copied to the database at a time. If after triggering the replication of the Memtable to the database, more first data continues to be written in the memory, and the first data written later does not reach the preset amount of data, then the first Memtable written can be copied After reaching the database, when the next Memtable replication opportunity is reached, it is triggered to execute the method provided by the embodiment of the present invention again.

In the process of copying the Memtable to the database, the Memtable is copied to obtain the Memtable' by means of replication. The Memtable remains in memory, and the Memtable' is transferred. In the process of data transfer, this part of the data cannot be read either in memory or in the database. However, according to the embodiment of the present invention, since the Memtable is still reserved in the memory, the data read and write transactions can be executed by reading the Memtable in the memory, without affecting the normal execution of the data read and write transactions.

In order to save the storage space of the database, optionally, the above process of copying the Memtable to the database can also be implemented as: compressing the Memtable; and copying the compressed Memtable to the database.

After the data replication operation, the same data exists in the memory and the database, that is, the replicated Memtable. In order to avoid a transaction execution error caused by repeatedly reading the same data when executing a data read/write transaction, the data may be distinguished by the first record information and the second record information. The first record information or the second record information includes identification information of the operation data required when executing the data read and write transaction.

The functions of the first record information and the second record information will be described below in conjunction with the specific process of executing the data read/write transaction.

According to the first record information and the second record information, the process of performing the data read and write transaction processing can be implemented as: receiving the data read and write transactions, according to the receiving time of the data read and write transactions corresponding to the first record information and the second record information respectively. Generation time, which identifies the reference record information that matches the data read and write transactions. Execute a data read/write transaction based on the reference record information, that is, determine the data to be accessed to execute the data read/write transaction according to the identification information contained in the reference record information, so as to execute the data read/write transaction based on the data. The reference record information is the first record information or the second record information.

Before executing the data read/write transaction based on the reference record information, it is necessary to select the reference record information matching the data read/write transaction for the data read/write transaction among the first record information and the second record information. The process of selecting the reference record information can be implemented as follows: if the reception time of the data read-write transaction is before the generation time of the second record information, the reference record information is determined to be the first record information; After the generation time of the second record information, the reference record information is determined to be the second record information.

In a practical application, it is assumed that a data read/write transaction is received at a certain time, and then the receiving time of the data read/write transaction can be compared with the generation time of the second record information. If the reception time of the data read/write transaction is before the generation time of the second record information, the first record information can be selected as the reference record information, if the reception time of the data read/write transaction is after the generation time of the second record information, Then the second record information can be selected as the reference record information. It can be understood that since the generation time of the second record information can be obtained, it means that when the second record information is generated, the second record information and the corresponding generation time can be stored, and the subsequent generation of the second record information can be determined based on this. time.

After determining the reference record information that matches the data read/write transaction, you can determine the storage location of the data required to perform the data read/write transaction, and then you can go to these storage locations to query the data. Data processing, etc., to complete data read and write transactions.

For ease of understanding, the following describes the process of executing data read and write transactions with reference to specific examples and FIG. 2 .

In Figure 2, at time 1, there is 1 Memtable' in the memory, marked as M0, and there are 3 data that have been transferred from the memory in the database, marked as R0, R1 and R2. Before transferring M0 to the database, record information A: M0, R0, R1 and R2 can be generated. Next, M0 can be replicated, assuming that the replicated data is marked as R3. Subsequently, the R3s can be moved into the database, thus increasing the R3s in the database. At time 2 when the copy operation is completed, record information B: R0, R1, R2, and R3 can be generated. If the data read/write transaction X is received at a certain time, it can be judged whether the reception time of the data read/write transaction X is after time 2, and if the reception time of the data read/write transaction X is before time 2, use the record information A to execute the data read Write transaction X, that is, query the data for the desired operation in M0, R0, R1 and R2. If the reception time of the data read/write transaction X is after time 2, use the record information B to execute the data read/write transaction X, that is, query the data required for the operation in R0, R1, R2 and R3. Regardless of whether the record information A or the record information B is used to execute the data read and write transaction X, it can be ensured that there is no repeated query of data and that the queried data is all stored data, thus avoiding the occurrence of data missing.

In order to reduce the occupation of storage space, optionally, if the data read and write transactions performed based on the first record information (that is, the data read and write transactions triggered before the second record information is generated) have been submitted, the first record information can be deleted. and the Memtable in the memory corresponding to the first record information.

In some cases, multiple records may exist at the same time. For example, if the rate of copying a Memtable to the database is slower than the rate at which a new Memtable is generated, multiple Memtables will accumulate in the memory, and multiple Memtables will be queued to be copied to the database in turn, and the phenomenon of multiple Memtables may accumulate. Raises the presence of more than one record information at the same time. Then, the valid time of any record information can be considered as: from the generation of the record information to the generation of the next record information, the data read and write transactions received during this period will use the record information.

With the method provided by the embodiment of the present invention, in the process of storing the data in the memory to the database, the Memtable is kept in the memory, and only the Memtable' is transferred, so that the Memtable can still be queried in the memory, avoiding the need for The problem of missing data during data transfer. In addition, in the method provided by the embodiment of the present invention, when a data read/write transaction is received, the corresponding record information may be used to perform data read/write based on the reception time of the data read/write transaction, so as to complete the transaction submission.

In the following, in conjunction with the embodiment shown in FIG. 3 , a solution for copying the Memtable to the database is exemplarily described. As shown in Figure 3, the replication scheme may include the following steps:

301. Perform storage format conversion on the first data of a preset data amount.

302. Copy the data converted in the storage format into the database.

In practical applications, if the storage format of the data in the memory is inconsistent with the storage format of the data in the database, the storage format conversion can be performed on the Memtable' first, so as to convert the Memtable' into data that conforms to the storage format in the database, and then Copy the converted Memtable' into the database.

In practical applications, the database may include a row database, a column database, and the like. The columnar database may be, for example, a Histore database. If the database is a columnar database, and the storage format of the data in the columnar database is different from the storage format of the data in the memory, the storage format of the Memtable' can be converted, and then the Memtable' after the storage format conversion can be stored in the columnar database. middle.

The following will introduce the storage format of the data in the columnar database and the storage format of the data in the memory, and how to convert the storage format.

In practical applications, data is stored in a row in memory, and data is stored in a column in a columnar database. Suppose Memtable' includes multiple rows of data, and each row of data includes attribute values corresponding to multiple attributes. These data are stored row by row in memory, that is, multiple attribute values of the same row of data are closely arranged together as stored as a whole.

For ease of understanding, let’s take the example of entering students’ math test scores into the database. Suppose there are 10 students in a class, and now create a student transcript based on the math test results of the 10 students. The student’s transcript includes a total of 10 students. Rows of data, each row of data corresponds to a student. For each student, his name, student number, class, and math grades need to be recorded on the student transcript. Correspondingly, name, student ID, class, and math grade can be used as attributes described in this embodiment, and specific contents corresponding to name, student ID, class, and math grade can be used as attribute values. For example, the attribute value corresponding to the name can be student A, the attribute value corresponding to the student ID can be 20200114, the attribute value corresponding to the class can be 2nd grade 3rd class, and the math score is 90, then this row of data is (student A, 20200114, Grade 2, Class 3, 90).

For the multiple rows of data contained in the above Memtable', when converting the storage format, multiple attribute values corresponding to the same attribute in the multiple rows of data are combined to obtain multiple data blocks, and the multiple data blocks are combined. stored in a columnar database.

For a more intuitive understanding of the storage format conversion process, an exemplary description is given with reference to FIG. 4 .

In the memory, the data is stored in the storage format of [name, student ID, class, math grade], that is, a piece of data contains attribute values of multiple attributes. The memory shown in Figure 4 actually includes two rows of data: the first row of data is: [Student A, 20200114, Grade 2, Class 3, 90], and the second row of data is: [Student B, 20200115, Grade 2, Class 3 , 95]. When converting the storage format, you can combine student A and student B corresponding to the name to obtain data block a; combine 20200114 and 20200115 corresponding to the student number to obtain data block b; Combine 2nd grade 3 and 2nd grade 3 to get data block c; combine 90 and 95 corresponding to math grades to get data block d.

After the multiple data blocks are obtained, the respective storage locations of the multiple data blocks in the database can be determined, and the multiple data blocks are stored in the respective corresponding storage locations.

In practical applications, each storage location can store one data block, and the storage locations corresponding to multiple data blocks are located in the same row. Multiple data blocks located in the same row can be called a row group (Rowgroup).

Although a row group includes a plurality of data blocks, wherein each data block corresponds to a different attribute, a plurality of data blocks in a row group corresponds to N rows of complete data. At the same time, by arranging multiple data blocks in the same row, it can be ensured that the same column of data blocks of multiple row groups corresponds to the same attribute.

The process of copying the Memtable to the database is exemplarily described below with reference to FIG. 5 . Assuming that there is currently a data table, the data table has written 1017 rows of data, of which 0-999 rows of data have been stored in the database, and correspond to row group 0, row group 1, row group 2 and row in the database Multiple data blocks are shown in group 3, where each row group contains four data blocks. Suppose 1000-1017 rows of data are stored in memory. In memory, it is assumed that every 6 rows of data constitute a Memtable, so there are a total of 3 Memtables in memory. The first Memtable corresponds to rows 1000-1005, the second Memtable corresponds to rows 1006-1011, and the third Memtable corresponds to rows 1012-1017. The three Memtables need to be copied to the database in turn. The copying order is the first Memtable, the second Memtable, and the third Memtable. Taking the replication process of the first Memtable as an example, the storage format of the first Memtable can be converted first, assuming that four data blocks are obtained, including data block a, data block b, data block c, and data block d. At this time, data block a, data block b, data block c, and data block d can be stored in the database as row group 4, and it is ensured that data blocks corresponding to the same attribute are arranged in the same column.

Based on this, in a columnar database, the data blocks arranged in the same column have the same attributes. For example, the data in the data block in the fourth column are all students' math scores. In this way, in some data read and write scenarios, it is convenient to query the data required for operations when executing data read and write transactions.

In order to further improve the efficiency of querying data, data statistics can be performed on each data block to obtain the data statistics information corresponding to each data block. The information is in the database, and some unused data blocks are filtered out and not scanned, and only the remaining data blocks are scanned. Optionally, after multiple data blocks are generated, the respective data statistics information of the multiple data blocks may be obtained, and the respective data statistics information corresponding to the multiple data blocks may be correspondingly stored in the respective storage locations of the multiple data blocks. .

The statistical information of the data may include statistical information such as the maximum value, the minimum value, the average value, and the mean square error in the data block. For any data block i in multiple data blocks, after obtaining the data block i, its maximum, minimum, average, mean square error and other statistical information can be calculated, and the data statistics corresponding to the data block i can be correspondingly Stored in the storage location corresponding to the data block i.

Based on this, the process of querying data can be specifically implemented as: in response to data read/write transactions corresponding to multiple data blocks, filtering out data blocks that do not satisfy the data read/write transactions according to data statistics information.

In the process of querying data, you can preferentially read the data statistics of each data block without reading the data inside each data block. If it is determined based on the data statistics corresponding to any data block that the data block does not meet the current data read requirements If the data block of the transaction is written, the data block can be filtered out, that is, the data inside the data block is not read.

Take querying students' math scores as an example. Suppose there are 4 columns of data blocks in the current database, and each column includes 5 data blocks. Among them, one column of data blocks corresponds to the attribute of mathematics scores, and the current data read and write transaction is the full score of the statistical mathematics test. (100 points) the number of students. At this point, you can first locate the column of data blocks whose attribute is the math score, assuming that the column of data blocks whose attribute is the math score is the fourth column of data blocks. Next, obtain the data statistics corresponding to each data block in the fourth column of data blocks to determine the maximum value in each of the data blocks in the fourth column of data blocks, such as 95, 88, 100, 99, 100. Based on this, it can be determined that the maximum value of the first data block, the second data block and the fourth data block in the data block in the fourth column has not reached the full score, so there is no mathematical perfect score in these three data blocks. Results, and then you can directly filter out these three data blocks, leaving the third data block and the fifth data block. Finally, you can read the math scores in the third data block and the fifth data block, and count how many perfect math scores there are.

It can be seen from the above example that when the data to be operated when executing a data read/write transaction corresponds to the same attribute, you can directly query the data block in the same column to execute the data read/write transaction, and the data query efficiency is high. However, if the data required to operate on a data read and write transaction corresponds to at least two attributes, it may be more advantageous to choose to store the data in rows. Taking the query of students' test scores as an example, suppose that the current data reading and writing transaction is the list of students who failed the statistical test. At this time, the data required to perform data reading and writing transactions includes not only the test scores, but also the corresponding students' data. Name corresponds to two attributes, and if the data is stored in rows, the student's name and test scores can be directly read. If the test scores are unqualified, the corresponding student's name can be directly output.

The row-based data block and the column-based database have been mentioned above. In practical applications, there is another kind of database that is a row-column hybrid database. In this row-column hybrid database, you can choose to store data in rows or columns. If you choose to store data in rows, the data storage format may not be converted during the process of copying data from memory to a row-column hybrid database. If you choose to store data in columns, the data storage format needs to be converted during the process of copying data from memory to a row-column hybrid database.

The storage format in which the data is stored in the row-column hybrid database, or whether data storage format conversion needs to be performed before the data is stored, can be determined according to the query feature information of the service corresponding to the first data. Optionally, before transferring the Memtable to the database, the query feature information of the service corresponding to the Memtable can be determined; according to the query feature information, it is determined whether to perform storage format conversion on the Memtable.

In practical applications, the data in a Memtable often comes from the same service, and different services often have different query feature information. In practical applications, for the data read transactions of a service, the characteristics of the data read transactions can be counted. For example, the data required to perform most of the data read transactions correspond to the same attribute, or most The data to be manipulated in the data read transaction corresponds to different attributes. Based on this, when the data required to perform most of the data read transactions corresponds to the same attribute, it can be considered that the data of the service is suitable for storage in rows. When most data read transactions are performed, the required operations are When the data corresponds to different attributes, it can be considered that the data of the service is suitable for storing in columns.

It can be understood that the corresponding relationship between the service and the query feature information can be established. Before transferring the Memtable to the database, the service to which the Memtable belongs can be determined, and then the query feature information corresponding to the service to which the Memtable belongs can be determined based on the above corresponding relationship. After the query feature information is determined, it may be determined whether to perform storage format conversion on the Memtable according to the query feature information. If it is determined that the storage format conversion of the Memtable is required, the storage format conversion of the Memtable can be performed. If it is determined that the storage format conversion of the Memtable is not required, the Memtable can be directly transferred to the database.

With the method provided by the embodiment of the present invention, the data stored in the row in the internal dimension can be converted into the data block stored in the column as needed, and the converted data block can be stored in the column data block. Improve the efficiency of data query.

The data storage device of one or more embodiments of the present invention will be described in detail below. Those skilled in the art can understand that these data storage devices can be configured by using commercially available hardware components through the steps taught in this solution.

FIG. 6 is a schematic structural diagram of a data storage device according to an embodiment of the present invention. As shown in FIG. 6 , the device includes: a writing module 601 , a generating module 602 , and a copying module 603 .

The writing module 601 is configured to write first data into the memory, where the first data is data to be stored in the database.

The generation module 602 is used for generating first record information when the first data written in the memory reaches a preset data amount, the first record information includes first identification information and second identification information, and the first identification information is The identification information corresponding to the first data of the preset data amount in the memory, and the second identification information is the identification information corresponding to the second data stored in the database in the database.

The copying module 603 is configured to copy the first data of the preset data amount to the database.

The generating module 602 is configured to generate second record information, where the second record information includes second identification information and third identification information corresponding to the first data of the preset data volume, and the third identification information is the third identification information of the preset data volume. Identification information corresponding to a data in the database.

The processing module 604 processes data read and write transactions according to the first record information and the second record information.

Optionally, the copying module 603 is specifically configured to: perform storage format conversion on the first data of the preset data amount; and copy the data after the storage format conversion into the database.

Optionally, the first data of the preset data volume includes multiple rows of data, each row of data includes attribute values corresponding to multiple attributes, and the copying module 603 is further configured to: copy the attribute values corresponding to the same attribute in the multiple rows of data. Combine to obtain multiple data blocks; determine the respective storage locations of multiple data blocks in the database, and the storage locations corresponding to multiple data blocks are located in the same row; store multiple data blocks in their respective corresponding storage locations.

Optionally, the copying module 603 is further configured to: compress the first data of the preset data amount; and copy the compressed data to the database.

Optionally, the device further includes: a statistics module, configured to obtain the data statistics information corresponding to the multiple data blocks; and store the data statistics information corresponding to the multiple data blocks in the storage corresponding to the multiple data blocks. position.

Optionally, the apparatus further includes: a filtering module configured to, in response to data read/write transactions corresponding to multiple data blocks, filter out data blocks that do not satisfy the data read/write transactions according to data statistics information.

Optionally, the device further includes: a determination module configured to determine query feature information of a service corresponding to the first data of the preset data volume; according to the query feature information, determine whether to perform a query on the first data of the preset data volume. Storage format conversion.

Optionally, the processing module 604 is specifically configured to: receive a data read/write transaction; determine the corresponding generation time of the first record information and the second record information according to the reception time of the data read/write transaction and the corresponding generation time of the second record information. The reference record information matched with the data read and write transaction, the reference record information is the first record information or the second record information; based on the identification information contained in the reference record information, the data to be accessed to execute the data read and write transaction is determined.

Optionally, the transaction processing module is specifically configured to: if the reception time of the data read/write transaction is before the generation time of the second record information, then determine that the reference record information is the first record information; if the reception time of the data read/write transaction is the first record information; After the generation time of the second record information, the reference record information is determined to be the second record information.

Optionally, the reference record information is the first record information, and the transaction processing module is further configured to delete the first record information and the first data of the preset data amount in the memory when the data read/write transaction is submitted.

The apparatus shown in FIG. 6 can execute the data storage methods provided in the embodiments shown in FIG. 1 to FIG. 5 . For the detailed execution process and technical effects, refer to the descriptions in the foregoing embodiments, which will not be repeated here.

In a possible design, the structure of the data storage device shown in FIG. 6 may be implemented as a server. As shown in FIG. 7 , the server may include: a processor 701 and a memory 702 . The memory 702 stores executable codes, and when the executable codes are executed by the processor 701, the processor 701 can at least implement the steps provided in the embodiments shown in FIG. 1 to FIG. 5. data storage method.

Optionally, the server may further include a communication interface 703 for communicating with other devices.

In addition, an embodiment of the present invention provides a non-transitory machine-readable storage medium, where executable codes are stored on the non-transitory machine-readable storage medium, and when the executable codes are executed by a processor of a server, The processor is made to implement at least the data storage method provided in the embodiments shown in the aforementioned FIG. 1 to FIG. 5 .

The apparatus embodiments described above are merely illustrative, wherein units described as separate components may or may not be physically separate. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and certainly can also be implemented by combining hardware and software. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of computer products in essence or that contribute to the prior art. In the form of a computer program product embodied on a medium (including but not limited to disk storage, CD-ROM, optical storage, etc.).

The data storage method provided by the embodiment of the present invention may be executed by a certain program/software, the program/software may be provided by the network side, and the electronic device mentioned in the foregoing embodiment may download the program/software to a local non-easy In the volatile storage medium, and when it needs to execute the foregoing data storage method, the program/software is read into the memory by the CPU, and then the program/software is executed by the CPU to realize the data storage method provided in the foregoing embodiment. , and the execution process may refer to the foregoing schematic diagrams in FIG. 1 to FIG. 5 .

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (13)

1. A data storage method, the method comprising: Write first data into the memory, the first data is the data to be stored in the database; If the first data written in the memory reaches a preset amount of data, first record information is generated, and the first record information includes first identification information and second identification information, and the first identification information is The identification information corresponding to the first data of the preset data volume in the memory, and the second identification information is the identification information corresponding to the second data stored in the database in the database; copying the first data of the preset data amount to the database; generating second record information, the second record information includes the second identification information and third identification information corresponding to the first data of the preset data volume, and the third identification information is the preset Identification information corresponding to the first data of the data volume in the database; Receive a data read-write transaction, and determine the data to be accessed to execute the data read-write transaction according to the reception time of the data read-write transaction and the respective generation times of the first record information and the second record information; According to the data to be accessed, the data read and write transactions are processed. 2. The method according to claim 1, wherein the copying of the first data of the preset data amount to the database comprises: performing storage format conversion on the first data of the preset data amount; Copy the data converted into the storage format into the database. 3 . The method according to claim 2 , wherein the first data of the preset data amount comprises multiple rows of data, each row of data includes attribute values corresponding to a plurality of attributes, and the The first data is subjected to storage format conversion, including: combining attribute values corresponding to the same attribute in the multiple rows of data to obtain multiple data blocks; Copying the data converted into the storage format into the database includes: determining the respective storage locations of the multiple data blocks in the database, and the storage locations corresponding to the multiple data blocks are located in the same row; The plurality of data blocks are respectively stored in their corresponding storage locations. 4. The method of claim 3, further comprising: Acquiring respective data statistics information corresponding to the multiple data blocks; The data statistics information corresponding to each of the plurality of data blocks is correspondingly stored in the corresponding storage locations of the plurality of data blocks. 5. The method of claim 4, further comprising: In response to data read/write transactions corresponding to the plurality of data blocks, filter out data blocks that do not satisfy the data read/write transactions according to the data statistics information. 6. The method of claim 2, further comprising: determining the query feature information of the service corresponding to the first data of the preset data volume; According to the query feature information, it is determined whether to perform storage format conversion on the first data of the preset data amount. 7. The method according to claim 1, wherein the copying of the first data of the preset data amount to the database comprises: compressing the first data of the preset data amount; Copy the compressed data to the database. 8. The method according to any one of claims 1 to 7, wherein according to the reception time of the data read-write transaction and the generation time corresponding to the first record information and the second record information, determine The data to be accessed to execute the data read/write transaction includes: According to the reception time of the data read/write transaction and the respective generation times of the first record information and the second record information, the reference record information matching the data read/write transaction is determined, and the reference record information for the first record information or the second record information; Based on the identification information contained in the reference record information, the data to be accessed for executing the data read/write transaction is determined. 9. The method according to claim 8, wherein the determining of the reference record information matching the data read-write transaction comprises: If the reception time of the data read/write transaction is before the generation time of the second record information, determine that the reference record information is the first record information; If the reception time of the data read/write transaction is after the generation time of the second record information, the reference record information is determined to be the second record information. 10. The method according to claim 9, wherein the reference record information is the first record information, the method further comprising: If the data read/write transaction is committed, the first record information and the first data of the preset data amount in the memory are deleted. 11. A data storage device comprising: a writing module, configured to write first data into the memory, where the first data is the data to be stored in the database; A generating module is used to generate first record information when the first data written in the memory reaches a preset data amount, and the first record information includes first identification information and second identification information, and the The first identification information is the identification information corresponding to the first data of the preset data amount in the memory, and the second identification information is the identification corresponding to the second data stored in the database in the database information; a copying module, configured to copy the first data of the preset data amount to the database; The generating module is configured to generate second record information, the second record information includes the second identification information and the third identification information corresponding to the first data of the preset data volume, the third identification information The identification information is the identification information corresponding to the first data of the preset data volume in the database; a processing module, configured to receive a data read/write transaction, and determine to execute the data read/write transaction according to the reception time of the data read/write transaction and the corresponding generation time of the first record information and the second record information Data to be accessed; according to the data to be accessed, the data read and write transactions are processed. 12. A server, comprising: a memory and a processor; wherein, executable code is stored on the memory, and when the executable code is executed by the processor, the processor is caused to execute the method according to claim 1- The data storage method of any one of 10. 13. A non-transitory machine-readable storage medium having executable codes stored thereon, which, when executed by a processor of a server, cause the processor to execute The data storage method according to any one of claims 1-10.

Images