CN111694992B - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN111694992B CN111694992B CN201910198443.9A CN201910198443A CN111694992B CN 111694992 B CN111694992 B CN 111694992B CN 201910198443 A CN201910198443 A CN 201910198443A CN 111694992 B CN111694992 B CN 111694992B
- Authority
- CN
- China
- Prior art keywords
- data
- pointer
- target
- data processing
- hierarchy
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9027—Trees
Abstract
The present disclosure relates to a data processing method and apparatus. The method is applied to a data storage system based on an LSM-Tree structure, and comprises the following steps: receiving a data processing request containing a target keyword, wherein the data processing request is used for requesting to execute a data processing operation on target data corresponding to the target keyword; and executing data processing operation on the target data through a handle corresponding to the target key word, wherein the handle comprises a pointer for pointing to the target data. The access speed of the LSM-Tree structure can be improved, and further the data processing efficiency can be effectively improved.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method and apparatus.
Background
A log structured merge Tree (LSM-Tree, log Structured Merge Tree) is a structure widely used in the field of data storage. The LSM structure includes at least a high-level memory space (e.g., memtab) having a read-write property and a low-level memory space (e.g., baseTable) having a read-only property. When the amount of data in the MemTable exceeds a threshold, the LSM-Tree structure needs to merge the MemTable and the BaseTable to generate a new BaseTable, and to generate a new empty MemTable. The merging between memtab and BaseTable may cause a change in the memory resources used to store the data corresponding to the key. At present, when an LSM-Tree structure is accessed, resource positioning is carried out in the whole data storage system through keywords in each access, memory resources for storing data corresponding to the keywords are determined, and then data processing is carried out on the data in the memory resources, so that the data processing efficiency is low.
Disclosure of Invention
In view of this, the present disclosure proposes a data processing method and apparatus, so that the access speed of the LSM-Tree structure can be improved, and further, the data processing efficiency can be effectively improved.
According to a first aspect of the present disclosure, there is provided a data processing method applied to a data storage system based on an LSM-Tree structure, the method comprising: receiving a data processing request containing a target keyword, wherein the data processing request is used for requesting to execute a data processing operation on target data corresponding to the target keyword; and executing data processing operation on the target data through a handle corresponding to the target key word, wherein the handle comprises a pointer for pointing to the target data.
In one possible implementation, performing, by a handle corresponding to the target key, a data processing operation on the target data includes: judging whether the pointer is valid or not; and when the pointer is determined to be valid, performing a data processing operation on the target data through the pointer.
In one possible implementation, the method further includes: when the pointer is determined to be invalid, performing resource positioning in the data storage system according to the target keyword, and searching the target data; reassigning the pointer according to the search result; and executing data processing operation on the target data through the re-assigned pointer.
In one possible implementation, the LSM-Tree structure includes at least a first hierarchy and a second hierarchy, the data in the first hierarchy having a read-write characteristic, the data in the second hierarchy having a read-only characteristic; the method further comprises the steps of: determining a hierarchy of the data table as a third hierarchy when the data amount of the data table in the first hierarchy is greater than a threshold value, and generating a new empty data table and setting the hierarchy of the empty data table as the first hierarchy, wherein the data in the third hierarchy has a read-only characteristic; data in the third hierarchy is merged to the second hierarchy.
In one possible implementation, the pointers include a read pointer and a write pointer; the method further comprises the steps of: setting the write pointer to a null pointer when a table of data in the first hierarchy is null and data in the third hierarchy is not merged to the second hierarchy; when the data in the third hierarchy is merged after the second hierarchy and the write pointer is a null pointer, the read pointer is set to a null pointer.
In one possible implementation, determining whether the pointer is valid includes at least one of: when the read pointer is a non-null pointer, determining that the read pointer is valid; and when the write pointer is a non-null pointer, determining that the write pointer is valid.
In a possible implementation manner, the handle further includes a first identifier, the LSM-Tree structure includes a second identifier, and before the data in the third hierarchy is merged into the second hierarchy, the first identifier and the second identifier are the same; the method further comprises the steps of: and after the data in the third hierarchy is merged into the second hierarchy, updating the second identifier, wherein the updated second identifier is different from the first identifier.
In one possible implementation, the pointers include a read pointer and a write pointer; judging whether the pointer is valid or not, comprising: and when the first identifier and the second identifier are the same, determining that the read pointer and the write pointer are valid.
In one possible implementation, the method further includes: and after resource positioning is carried out in the data storage system according to the target keyword, updating the first identifier, wherein the updated first identifier is the same as the second identifier.
In one possible implementation, when the pointer is determined to be valid, performing, by the pointer, a data processing operation on the target data, including at least one of: reading the target data pointed by the read pointer; and modifying the target data pointed by the write pointer.
In one possible implementation, reassigning the pointer according to the search result includes: if the target data is found, the memory address of the target data is assigned to the read pointer.
In one possible implementation, performing, by the reassigned pointer, a data processing operation on the target data includes: and reading the target data pointed by the read pointer after reassignment.
In one possible implementation, reassigning the pointer according to the search result includes: if the target data is found, copying the target data to the first level; and assigning the memory address of the target data in the first hierarchy to the write pointer.
In one possible implementation, performing, by the reassigned pointer, a data processing operation on the target data includes: and modifying the target data pointed by the re-assigned write pointer.
In one possible implementation, reassigning the pointer according to the search result includes: if the target data is not found, creating an empty memory resource in the first level; and assigning the memory address of the empty memory resource to the write pointer.
In one possible implementation, performing, by the reassigned pointer, a data processing operation on the target data includes: and writing the target data in the empty memory resource pointed by the re-assigned write pointer.
According to a second aspect of the present disclosure, there is provided a data processing apparatus for application to a data storage system based on an LSM-Tree structure, the apparatus comprising: a receiving module, configured to receive a data processing request including a target keyword, where the data processing request is used to request to perform a data processing operation on target data corresponding to the target keyword; and the data processing module is used for executing data processing operation on the target data through a handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data.
According to a third aspect of the present disclosure, there is provided a data processing apparatus comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the data processing method of the first aspect.
According to a fourth aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer program instructions, wherein the computer program instructions, when executed by a processor, implement the data processing method of the first aspect described above.
In a data storage system based on an LSM-Tree structure, receiving a data processing request containing a target keyword, the data processing request being for requesting execution of a data processing operation on target data corresponding to the target keyword; and further, executing data processing operation on the target data through a handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data. Setting a handle corresponding to the keyword in the LSM-Tree structure, and directly accessing the data corresponding to the keyword in the LSM-Tree structure through the handle corresponding to the keyword to execute data processing operation, so that the access speed of the LSM-Tree structure can be improved, and the data processing efficiency can be further effectively improved.
Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments, features and aspects of the present disclosure and together with the description, serve to explain the principles of the disclosure.
FIG. 1 illustrates a schematic diagram of a data storage system based on an LSM-Tree architecture;
FIG. 2 shows a flow diagram of a data processing method of an embodiment of the present disclosure;
FIG. 3 shows a schematic diagram of an LSM-Tree structure of an embodiment of the present disclosure;
FIG. 4 shows a schematic diagram of data merging in an LSM-Tree structure in accordance with an embodiment of the present disclosure;
FIG. 5 illustrates a schematic diagram of pointers in an update handle of an embodiment of the present disclosure;
fig. 6 is a diagram of a second sequence number in an updated LSM-Tree structure according to an embodiment of the present disclosure;
FIG. 7 is a schematic diagram showing a structure of a data processing apparatus according to an embodiment of the present disclosure;
fig. 8 shows a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
Various exemplary embodiments, features and aspects of the disclosure will be described in detail below with reference to the drawings. In the drawings, like reference numbers indicate identical or functionally similar elements. Although various aspects of the embodiments are illustrated in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated.
The word "exemplary" is used herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.
In addition, numerous specific details are set forth in the following detailed description in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order not to obscure the present disclosure.
In a data storage system based on an LSM-Tree structure, a storage format of data is a key-value structure, value is a data value of the data, and key value (key) is an index parameter of the data. FIG. 1 shows a schematic diagram of a data storage system based on an LSM-Tree architecture. In the data storage system shown in fig. 1, the LSM-Tree structure includes a data table MemTabel (Memory Table) for storing data in a first level (high level) and a data table BaseTable for storing data in a second level (low level). Wherein, the MemTable has a read-write property and the BaseTable has a read-only property. When key-value data is written, the Memtab in the first hierarchy is written first. When the data quantity of the MemTable in the first level exceeds a threshold value, merging the MemTable with the BaseTable in the second level to obtain a new BaseTable in the second level, and generating a new empty MemTale in the first level to continue data writing.
The data written into the LSM-Tree structure is stored in a layered manner, the data with the same key words can be stored in data tables of different levels, and the data stored in the data tables of higher levels are new data relative to the data stored in the data tables of lower levels. That is, if the data with the same key exists in the MemTable and the BaseTable, the data corresponding to the key in the MemTable is new data with respect to the data corresponding to the key in the BaseTable, and when the MemTable and the BaseTable are merged, the data corresponding to the key in the MemTable covers the data corresponding to the key in the BaseTable.
As stated in the background section, since merging of MemTable and BaseTable may cause a change in a memory address storing data corresponding to a key, when accessing the LSM-Tree structure, resource location needs to be performed according to the key each time, a memory address storing data corresponding to the key is determined, and then the memory address is released after performing data processing operations such as a read operation or a write operation according to the memory address. The access mode of each time of resource positioning according to the keywords is limited by the scale of the data storage system, when the scale of the data storage system is large, the access speed of the LSM-Tree structure is low, and the data processing efficiency is reduced.
The data processing method provided by the disclosure can be applied to the data storage system based on the LSM-Tree structure shown in FIG. 1 to improve the access speed of the LSM-Tree structure, and further effectively improve the data processing efficiency. The data processing method provided by the present disclosure is described in detail below.
FIG. 2 shows a flow diagram of a data processing method of an embodiment of the present disclosure. The method shown in fig. 2 may be applied to a data storage system based on an LSM-Tree structure, and may include:
Step 21, receiving a data processing request including a target keyword, the data processing request being for requesting execution of a data processing operation on target data corresponding to the target keyword.
In step 22, a data processing operation is performed on the target data by means of a handle corresponding to the target key, the handle including a pointer for pointing to the target data.
A global handle linked list is created in the LSM-Tree structure, and all handles created in the LSM-Tree structure are stored in the global handle linked list. Each handle includes a key and a pointer for pointing to data corresponding to the key, so that data processing operation can be directly performed on the data corresponding to the key through the handle corresponding to the key without performing resource positioning.
In one possible implementation, the pointers in the handle include a read pointer and a write pointer.
Fig. 3 shows a schematic diagram of an LSM-Tree structure of an embodiment of the present disclosure. As shown in fig. 3, the LSM-Tree structure includes a first level (high level) memtab, a second level (low level) BaseTable, and a global handle linked list. The global handle chain table comprises a handle A corresponding to the first key and a handle B corresponding to the second key. Memtab includes: data a corresponding to the first key and data b corresponding to the second key. The BaseTable includes: data c corresponding to the first key and data d corresponding to the third key. The handle a includes: the data processing method comprises a first key word, a read pointer and a write pointer, wherein the read pointer and the write pointer in a handle A both point to data a corresponding to the first key word in the Memcabel, namely the data a can be directly accessed through the read pointer and the write pointer in the handle A, and then data processing operation is executed on the data a. The handle B includes: the second key word, the read pointer and the write pointer, wherein the read pointer and the write pointer in the handle B both point to the data B corresponding to the second key word in the Memcabel, namely the data B can be directly accessed through the read pointer and the write pointer in the handle B, and then the data processing operation is executed on the data B.
In one possible implementation manner, a handle corresponding to the hot data, that is, a handle corresponding to a keyword of the hot data, may be created for hot data with higher access rate in the LSM-Tree structure, so as to improve the access speed of the hot data.
In addition to creating handles for hot data in the LSM-Tree structure, handles may be created for other data in the LSM-Tree structure according to actual needs, which is not specifically limited in this disclosure.
In one possible implementation, the LSM-Tree structure includes at least a first level and a second level, the data in the first level having a read-write characteristic, the data in the second level having a read-only characteristic; determining a hierarchy of the data table as a third hierarchy when the data amount of the data table in the first hierarchy exceeds a threshold, and generating a new empty data table and setting the hierarchy of the empty data table as the first hierarchy, wherein data in the third hierarchy has a read-only characteristic; data in the third hierarchy is merged to the second hierarchy.
The threshold may be set according to actual circumstances, which is not particularly limited in this disclosure.
Taking fig. 3 as an example, if the amount of data in memtab of fig. 3 exceeds the threshold, then the data merge shown in fig. 4 needs to be performed. Fig. 4 shows a data merge diagram in an LSM-Tree structure of an embodiment of the present disclosure. As shown in fig. 4, the hierarchy of memtab whose data amount exceeds the threshold is set from the first hierarchy to a third hierarchy, i.e., the memtab whose data amount exceeds the threshold is changed to a data table ImmuTable (Immutable Table) in the third hierarchy, and a new empty memtab is generated in the first hierarchy for subsequent data writing, wherein the third hierarchy is located between the first hierarchy and the second hierarchy, and the data in the third hierarchy has a read-only property.
And merging the data of the ImmuTable and the BaseTable to obtain a new BaseTable in the second hierarchy. Because the data a corresponding to the first keyword in the imutable is new data relative to the data c corresponding to the first keyword in the BaseTable, the data a corresponding to the first keyword in the imutable covers the data c corresponding to the first keyword in the BaseTable in the data merging process. The new BaseTable obtained includes: data a corresponding to the first keyword, data b corresponding to the second keyword, and data d corresponding to the third keyword.
In one possible implementation, performing a data processing operation on target data by a handle corresponding to a target key includes: judging whether the pointer is valid or not; when the pointer is determined to be valid, a data processing operation is performed on the target data by the pointer.
In one possible implementation, the method further includes: when the pointer is determined to be invalid, performing resource positioning in the data storage system according to the target key words, and searching target data; reassigning the pointer according to the search result; and executing data processing operation on the target data through the reassigned pointer.
When the pointer in the handle corresponding to the target key is effective, the fact that data combination does not occur in the LSM-Tree structure is indicated, and the target data can be directly accessed through the pointer and data processing operation is executed; when the pointer in the handle corresponding to the target key is invalid, the pointer indicates that data merging occurs in the LSM-Tree structure, the memory address of the data corresponding to the target key is changed, the pointer cannot accurately point to the target data, and at the moment, after resource positioning searching for the target data is needed to be carried out again according to the target key, the pointer is reassigned, and then the target data can be accessed and data processing operation is executed through the pointer.
The ways of determining whether the pointer is valid include, but are not limited to, the following two ways.
First kind:
in one possible implementation, when the data table in the first hierarchy is empty and the data in the third hierarchy is not merged into the second hierarchy, the write pointer is set to a null pointer; when the data in the third hierarchy is merged into the second hierarchy and the write pointer is a null pointer, the read pointer is set to the null pointer.
In one possible implementation, determining whether the pointer is valid includes at least one of: when the read pointer is a non-null pointer, determining that the read pointer is valid; when the write pointer is a non-null pointer, it is determined that the write pointer is valid.
And in the process of executing the data merging of the ImmuTable and the BaseTable in the LSM-Tree structure, synchronously updating the pointer in each handle in the global handle chain table.
After the memtab of which the data quantity exceeds the threshold value is changed to the ImmuTable and a new empty memttable is generated and before the ImmuTable and the BaseTable are combined, traversing the global handle linked list at the moment, setting a write pointer in each handle in the global handle linked list as an empty pointer, and keeping the read pointer unchanged because the ImmuTable has read-only property. I.e. for either handle, the data access and data processing operations can only be performed directly by the read pointer in the handle, but not by the write pointer in the handle.
After the data combination is carried out on the ImmuTable and the BaseTable, the memory address of the data is changed, at the moment, the global handle chain table is traversed, and any handle in the global handle chain table is aimed at: if the write pointer in the handle is still a null pointer, the fact that resource positioning is not performed on the write pointer through the keywords in the handle in the LSM-Tree structure is indicated, and at the moment, the read pointer in the handle is set to be the null pointer, namely, data access and data processing operation cannot be directly performed through the read pointer and the write pointer in the handle; if the write pointer in the handle is a non-null pointer, the fact that the resource location is carried out on the write pointer through the key words in the handle in the LSM-Tree structure is indicated to carry out re-assignment on the write pointer, and at the moment, data access and data processing operation can be directly carried out through the read pointer and the write pointer in the handle without updating the read pointer and the write pointer in the handle.
Taking fig. 4 as an example, fig. 5 is a schematic diagram illustrating a pointer in an update handle in the process of performing data merging shown in fig. 4 according to an embodiment of the disclosure. As shown in fig. 5, before the memtab of which the data amount exceeds the threshold value is changed to ImmuTable, and a new empty memttable is generated, and the ImmuTable and the BaseTable are subjected to data merging, traversing the global handle linked list, setting the write pointers in the handle a and the handle B as null pointers, and keeping the read pointers unchanged.
The write pointer in handle a is set to a null pointer indicating that the write pointer is invalid, i.e. the data a in the ImmuTable cannot be accessed and data processing operations performed by the write pointer in handle a, at this time, the data a in the ImmuTable can still be accessed and data processing operations performed by the read pointer in handle a. The write pointer in handle B is set to a null pointer indicating that the write pointer is invalid, i.e. the data B in the ImmuTable cannot be accessed and data processing operations performed by the write pointer in handle B, at this time, the data B in the ImmuTable can still be accessed and data processing operations performed by the read pointer in handle B.
After the data combination of the ImmuTable and the BaseTable, traversing the global handle chain table, and setting the read pointers in the handle A and the handle B as null pointers.
Both the read pointer and the write pointer in the handle a are set to null pointers, i.e., both the read pointer and the write pointer in the handle a are invalid, at which time neither the read pointer nor the write pointer in the handle a can directly access the data a in the BaseTable and perform data processing operations. The read pointer and the write pointer in the handle B are both set as null pointers, i.e. the read pointer and the write pointer in the handle B are both invalid, and at this time, neither the read pointer nor the write pointer in the handle B can directly access the data B in the BaseTable and perform data processing operations.
Because the pointers in each handle in the global handle chain table are synchronously updated in the process of executing the data merging of the ImmunTable and the BaseTable in the LSM-Tree structure, after a data processing request containing a target keyword is received, whether the pointers in the handles corresponding to the target keyword are effective or not can be determined by judging whether the pointers in the handles corresponding to the target keyword are non-null pointers, and when the pointers in the handles corresponding to the target keyword are determined to be effective, the target data is directly accessed and the data processing operation is executed through the pointers in the handles corresponding to the target keyword, and the target data is not required to be searched in the LSM-Tree structure through resource positioning, so that the access speed of the LSM-Tree structure is improved, and the data processing efficiency is further improved.
Second kind:
in one possible implementation, the handle further includes a first identifier, the LSM-Tree structure includes a second identifier, and before the data in the third hierarchy is merged into the second hierarchy, the first identifier and the second identifier are the same; and updating the second identifier after the data in the third hierarchy is merged into the second hierarchy, wherein the updated second identifier is different from the first identifier.
In one possible implementation, after resource location in the data storage system according to the target key, the first identifier is updated, and the updated first identifier is the same as the second identifier.
In one possible implementation, the pointers include a read pointer and a write pointer; determining whether the pointer is valid includes: and when the first identifier and the second identifier are the same, determining that the read pointer and the write pointer are valid.
In one possible implementation, the first identifier is a first serial number and the second identifier is a second serial number.
And in the process of executing the data merging of the ImmuTable and the BaseTable in the LSM-Tree structure, synchronously updating the second serial number in the LSM-Tree structure.
When the data quantity in the Memtab is smaller than the threshold value, or after the Memtab with the data quantity exceeding the threshold value is changed to ImmuTable and a new empty Memtab is generated, and before the ImmuTable and BaseTable are combined, the first sequence number in the handle is the same as the second sequence number in the LSM-Tree structure.
After the data merge of the ImmuTable and the BaseTable, the memory address of the data is changed, and the second sequence number is incremented, for example, the second sequence number is incremented by 1. At this time, the first serial number is different from the second serial number. After resource location is performed in the data storage system according to the key words in the handle and the pointers in the handle are reassigned, the first sequence number in the handle is updated so that the updated first sequence number in the handle is the same as the second sequence number in the LSM-Tree, namely, the pointers in the handle become valid.
Taking fig. 4 as an example, fig. 6 is a schematic diagram illustrating updating the second sequence number in the LSM-Tree in the process of performing the data merging shown in fig. 4 according to an embodiment of the disclosure. As shown in fig. 6, before memtab with data volume exceeding the threshold is changed to ImmuTable and new empty memttable is generated and data is merged with BaseTable, the first sequence number in handle a, the first sequence number in handle B and the second sequence number in LSM-Tree are 003.
After the data merge of the ImmuTable and the BaseTable, the second sequence number is incremented, and becomes 004.
The first sequence number 003 in the handle a is different from the second sequence number 004 in the LSM-Tree, that is, the read pointer and the write pointer in the handle a are both invalid, and at this time, the read pointer and the write pointer in the handle a cannot directly access the data a in the BaseTable and perform the data processing operation. The first sequence number 003 in the handle B is different from the second sequence number 004 in the LSM-Tree, that is, the read pointer and the write pointer in the handle B are both invalid, and at this time, the read pointer and the write pointer in the handle B cannot directly access the data B in the BaseTable and perform the data processing operation.
The first identifier and the second identifier may be other identifiers besides a serial number, which is not particularly limited in this disclosure.
Because the second identifier in the LSM-Tree structure is synchronously updated in the process of executing the data merging of the ImmunTable and the BaseTable in the LSM-Tree structure, after a data processing request containing a target keyword is received, whether the pointer in the handle corresponding to the target keyword is effective or not can be determined by judging whether the first identifier in the handle corresponding to the target keyword is identical with the second identifier in the LSM-Tree structure, when the pointer in the handle corresponding to the target keyword is determined to be effective, the target data is directly accessed and the data processing operation is executed through the pointer in the handle corresponding to the target keyword, and the target data corresponding to the target keyword is not required to be searched in the LSM-Tree structure through resource positioning, so that the access speed of the LSM-Tree structure is improved, and the data processing efficiency is further improved.
In one possible implementation, when the pointer is determined to be valid, performing a data processing operation on the target data by the pointer, including at least one of: reading target data pointed by the read pointer; the target data pointed to by the write pointer is modified.
When the read pointer in the handle corresponding to the target key is valid, the target data can be directly read through the read pointer; when the write pointer in the handle corresponding to the target key is valid, the target data can be directly modified through the write pointer.
When the pointers in the handles corresponding to the target keywords are determined to be invalid, performing resource positioning in the data storage system according to the target keywords, searching for target data, performing reassignment on the pointers in the handles corresponding to the target keywords according to the searching result, and further executing data processing operation on the target data through the reassigned pointers.
Performing resource positioning in a data storage system according to a target keyword, wherein the process of searching the target data comprises the following steps: step 1, searching target data in the Memable according to the target key word, if so, determining the memory address of the target data, and if not, continuing to execute the step 2; step 2, if the LSM-Tree structure contains an ImmuTable, executing the step 3, otherwise executing the step 4; step 3, searching target data in the ImmunTable according to the target keyword, if so, determining the memory address of the target data, and if not, continuing to execute the step 4; and step 4, searching target data in the BaseTable according to the target key, if so, determining the memory address of the target data, and if not, determining that the target data is not stored in the LSM-Tree structure.
In one possible implementation, reassigning the pointer according to the search result includes: if the target data is found, the memory address of the target data is assigned to the read pointer.
In one possible implementation, performing a data processing operation on the target data by reassigning the pointer includes: and reading the target data pointed by the read pointer after reassignment.
After the target data is found in the data storage system by performing resource positioning according to the target key, the memory address of the target data is assigned to the read pointer in the handle corresponding to the target key, so that before the next execution of merging of the data of the ImmuTable and the BaseTable in the LSM-Tree structure, the target data can be directly read through the read pointer in the handle corresponding to the target key without performing resource positioning again, and the access speed and the processing efficiency of the target data are effectively improved.
In one possible implementation, reassigning the pointer according to the search result includes: if the target data is found, copying the target data to a first level; and assigning the memory address of the target data in the first hierarchy to a write pointer.
In one possible implementation, performing a data processing operation on the target data by reassigning the pointer includes: and modifying the target data pointed by the reassigned write pointer.
Because only the MemTable has the writable characteristic, after the target data can be found in the data storage system according to the resource positioning of the target key, the target data is copied to the MemTable, and then the memory address of the target data in the MemTable is assigned to the write pointer in the handle corresponding to the target key, so that before the data combination of the ImmuTable and the BaseTable is executed next time in the LSM-Tree structure, the target data can be directly modified through the write pointer in the handle corresponding to the target key without carrying out resource positioning again, and the access speed and the processing efficiency of the target data are effectively improved.
In one possible implementation, reassigning the pointer according to the search result includes: if the target data is not found, creating an empty memory resource in the first level; and assigning the memory address of the empty memory resource to the read pointer and the write pointer.
After resource positioning is performed in the data storage system according to the target key, if the target data is not found, the fact that the target data is not stored in the data storage system is indicated, at this time, an empty memory resource is created in the memtab, and then the memory address of the empty memory resource is assigned to a read pointer and a write pointer in a handle corresponding to the target key, so that the target data is not stored in the data storage system can be directly determined according to the read pointer in the handle corresponding to the target key, and the resource positioning is not needed again, or the target data can be directly written into the memory resource according to the write pointer in the handle corresponding to the target key.
In one possible implementation, performing a data processing operation on the target data by reassigning the pointer includes: and writing target data in the empty memory resource pointed by the reassigned write pointer.
In a data storage system based on an LSM-Tree structure, receiving a data processing request containing a target keyword, the data processing request being for requesting execution of a data processing operation on target data corresponding to the target keyword; and further, executing data processing operation on the target data through a handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data. Setting a handle corresponding to the keyword in the LSM-Tree structure, and directly accessing the data corresponding to the keyword in the LSM-Tree structure through the handle corresponding to the keyword to execute data processing operation, so that the access speed of the LSM-Tree structure can be improved, and the data processing efficiency can be further effectively improved.
Fig. 7 shows a schematic structural diagram of a data processing apparatus according to an embodiment of the present disclosure. The apparatus 70 shown in fig. 7 may be applied to a data storage system based on an LSM-Tree structure, and may be used to perform the steps of the method embodiment shown in fig. 2 described above, the apparatus 70 comprising:
A receiving module 71 for receiving a data processing request including a target keyword, the data processing request being for requesting execution of a data processing operation on target data corresponding to the target keyword;
the data processing module 72 is configured to perform a data processing operation on the target data through a handle corresponding to the target keyword, where the handle includes a pointer for pointing to the target data.
In one possible implementation, the data processing module 72 includes:
the judging submodule is used for judging whether the pointer is effective or not;
and the data processing sub-module is used for executing data processing operation on the target data through the pointer when the pointer is determined to be valid.
In one possible implementation, the data processing module 72 further includes:
the searching sub-module is used for carrying out resource positioning in the data storage system according to the target key words when the pointer is determined to be invalid, and searching the target data;
the pointer assignment sub-module is used for reassigning the pointer according to the search result;
and the data processing sub-module is also used for executing data processing operation on the target data through the reassigned pointer.
In one possible implementation, the LSM-Tree structure includes at least a first level and a second level, the data in the first level having a read-write characteristic, the data in the second level having a read-only characteristic;
The apparatus 70 further comprises:
a generation module, configured to determine a level of the data table as a third level when the data amount of the data table in the first level exceeds a threshold, and generate a new empty data table and set the level of the empty data table as the first level, where data in the third level has a read-only characteristic;
and the merging module is used for merging the data in the third hierarchy to the second hierarchy.
In one possible implementation, the pointers include a read pointer and a write pointer;
the apparatus 70 further comprises:
the pointer updating module is used for setting the write pointer as a null pointer when the first-level total data table is null and the data in the third level is not merged into the second level;
the pointer updating module is further configured to set the read pointer to a null pointer when the data in the third hierarchy is merged into the second hierarchy and the write pointer is the null pointer.
In one possible implementation, the judging submodule includes:
a read pointer judging unit for determining that the read pointer is valid when the read pointer is a non-null pointer;
and the write pointer judging unit is used for determining that the write pointer is valid when the write pointer is a non-null pointer.
In one possible implementation, the handle further includes a first identifier, the LSM-Tree structure includes a second identifier, and before the data in the third hierarchy is merged into the second hierarchy, the first identifier and the second identifier are the same;
The apparatus 70 further comprises:
and the identifier updating module is used for updating the second identifier after the data in the third hierarchy is merged into the second hierarchy, and the updated second identifier is different from the first identifier.
In one possible implementation, the pointers include a read pointer and a write pointer;
the judging module is specifically configured to:
and when the first identifier and the second identifier are the same, determining that the read pointer and the write pointer are valid.
In one possible implementation manner, the identifier updating unit is further configured to update the first identifier after performing resource positioning in the data storage system according to the target keyword, where the updated first identifier is the same as the second identifier.
In one possible implementation, the data processing sub-module includes:
the reading unit is used for reading the target data pointed by the reading pointer;
and the modifying unit is used for modifying the target data pointed by the write pointer.
In one possible implementation, the pointer assignment module submodule is specifically configured to:
if the target data is found, the memory address of the target data is assigned to the read pointer.
In a possible implementation, the reading unit is further configured to:
and reading the target data pointed by the read pointer after reassignment.
In one possible implementation, the pointer assignment module submodule is specifically configured to:
if the target data is found, copying the target data to a first level;
memory addresses of the target data in the first hierarchy are assigned to the write pointer.
In a possible implementation, the modification unit is further configured to:
and modifying the target data pointed by the reassigned write pointer.
In one possible implementation, the pointer assignment module submodule is specifically configured to:
if the target data is not found, creating an empty memory resource in the first level;
and assigning the memory address of the empty memory resource to the write pointer.
In one possible implementation, the data processing sub-module further includes:
and the writing unit is used for writing the target data in the empty memory resource pointed by the reassigned writing pointer.
The apparatus 70 provided in the present disclosure can implement each step in the method embodiment shown in fig. 2, and achieve the same technical effects, and for avoiding repetition, a detailed description is omitted herein.
Fig. 8 shows a schematic structural diagram of an electronic device according to an embodiment of the present disclosure. As shown in fig. 8, at the hardware level, the electronic device comprises a processor, optionally together with an internal bus, a network interface, a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, network interface, and memory may be interconnected by an internal bus, which may be an ISA (Industry Standard Architecture ) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus, or EISA (Extended Industry Standard Architecture ) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in FIG. 8, but not only one bus or type of bus.
And a memory for storing the program. In particular, the program may include program code including computer-operating instructions. The memory may include memory and non-volatile storage and provide instructions and data to the processor.
The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs, forming the data processing device on a logic level. A processor executing the program stored in the memory, and specifically executing: receiving a data processing request containing a target keyword, wherein the data processing request is used for requesting to execute data processing operation on target data corresponding to the target keyword; the data processing operation is performed on the target data by a handle corresponding to the target key, the handle including a pointer for pointing to the target data.
In one possible implementation, the processor is specifically configured to perform: judging whether the pointer is valid or not; when the pointer is determined to be valid, a data processing operation is performed on the target data by the pointer.
In one possible implementation, the processor is specifically configured to perform: when the pointer is determined to be invalid, performing resource positioning in the data storage system according to the target key words, and searching target data; reassigning the pointer according to the search result; and executing data processing operation on the target data through the reassigned pointer.
In one possible implementation, the LSM-Tree structure includes at least a first level and a second level, the data in the first level having a read-write characteristic, the data in the second level having a read-only characteristic; the processor is specifically configured to perform: determining a hierarchy of the data table as a third hierarchy when the data amount of the data table in the first hierarchy exceeds a threshold, and generating a new empty data table and setting the hierarchy of the empty data table as the first hierarchy, wherein data in the third hierarchy has a read-only characteristic; data in the third hierarchy is merged to the second hierarchy.
In one possible implementation, the pointers include a read pointer and a write pointer; the processor is specifically configured to perform: when the data table in the first hierarchy is empty and the data in the third hierarchy is not merged into the second hierarchy, setting the write pointer as an empty pointer; when the data in the third hierarchy is merged into the second hierarchy and the write pointer is a null pointer, the read pointer is set to the null pointer.
In one possible implementation, the processor is specifically configured to perform: when the read pointer is a non-null pointer, determining that the read pointer is valid; when the write pointer is a non-null pointer, it is determined that the write pointer is valid.
In one possible implementation, the handle further includes a first identifier, the LSM-Tree structure includes a second identifier, and before the data in the third hierarchy is merged into the second hierarchy, the first identifier and the second identifier are the same; the processor is specifically configured to perform: and updating the second identifier after the data in the third hierarchy is merged into the second hierarchy, wherein the updated second identifier is different from the first identifier.
In one possible implementation, the pointers include a read pointer and a write pointer; the processor is specifically configured to perform: and when the first identifier and the second identifier are the same, determining that the read pointer and the write pointer are valid.
In one possible implementation, the processor is specifically configured to perform: and after the resource positioning is carried out in the data storage system according to the target keyword, updating the first identifier, wherein the updated first identifier is the same as the second identifier.
In one possible implementation, the processor is specifically configured to perform: reading target data pointed by the read pointer; the target data pointed to by the write pointer is modified.
In one possible implementation, the processor is specifically configured to perform: if the target data is found, the memory address of the target data is assigned to the read pointer.
In one possible implementation, the processor is specifically configured to perform: and reading the target data pointed by the read pointer after reassignment.
In one possible implementation, the processor is specifically configured to perform: if the target data is found, copying the target data to a first level; memory addresses of the target data in the first hierarchy are assigned to the write pointer.
In one possible implementation, the processor is specifically configured to perform: and modifying the target data pointed by the reassigned write pointer.
In one possible implementation, the processor is specifically configured to perform: if the target data is not found, creating an empty memory resource in the first level; and assigning the memory address of the empty memory resource to the write pointer.
In one possible implementation, the processor is specifically configured to perform: and writing target data in the empty memory resource pointed by the reassigned write pointer.
The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The various methods, steps and logic blocks disclosed in the embodiments of this specification may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present specification may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.
The electronic device may execute the method executed by the method embodiment shown in fig. 2 and implement the functions of the method embodiment shown in fig. 2, which are not described herein.
The present description also proposes a computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device comprising a plurality of application programs, enable the electronic device to perform the data processing method of the embodiment of fig. 2, and in particular to perform the steps of the method embodiment of fig. 2.
The present disclosure may be a system, method, and/or computer program product. The computer program product may include a computer readable storage medium having computer readable program instructions embodied thereon for causing a processor to implement aspects of the present disclosure.
The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.
The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.
Computer program instructions for performing the operations of the present disclosure can be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, c++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present disclosure are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information of computer readable program instructions, which can execute the computer readable program instructions.
Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the technical improvement of the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (17)
1. A data processing method, wherein the method is applied to a data storage system based on a log structured merge Tree LSM-Tree structure, the method comprising:
receiving a data processing request containing a target keyword, wherein the data processing request is used for requesting to execute a data processing operation on target data corresponding to the target keyword;
executing data processing operation on the target data through a handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data;
the executing, by the handle corresponding to the target keyword, a data processing operation on the target data includes:
Judging whether the pointer is valid or not;
performing a data processing operation on the target data by the pointer when the pointer is determined to be valid;
the method further comprises the steps of:
when the pointer is determined to be invalid, performing resource positioning in the data storage system according to the target keyword, and searching the target data;
reassigning the pointer according to the search result;
and executing data processing operation on the target data through the re-assigned pointer.
2. The method of claim 1, wherein the LSM-Tree structure comprises at least a first hierarchy and a second hierarchy, the data in the first hierarchy having read-write characteristics, the data in the second hierarchy having read-only characteristics;
the method further comprises the steps of:
determining a hierarchy of the data table as a third hierarchy when the data amount of the data table in the first hierarchy exceeds a threshold, and generating a new empty data table and setting the hierarchy of the empty data table as the first hierarchy, wherein the data in the third hierarchy has a read-only characteristic;
data in the third hierarchy is merged to the second hierarchy.
3. The method of claim 2, wherein the pointers comprise a read pointer and a write pointer;
The method further comprises the steps of:
setting the write pointer to a null pointer when a table of data in the first hierarchy is null and data in the third hierarchy is not merged to the second hierarchy;
when the data in the third hierarchy is merged after the second hierarchy and the write pointer is a null pointer, the read pointer is set to a null pointer.
4. A method according to claim 3, wherein determining whether the pointer is valid comprises at least one of:
when the read pointer is a non-null pointer, determining that the read pointer is valid;
and when the write pointer is a non-null pointer, determining that the write pointer is valid.
5. The method of claim 2, wherein the handle further comprises a first identifier, wherein the LSM-Tree structure comprises a second identifier, and wherein the first identifier and the second identifier are the same before the data in the third hierarchy is merged into the second hierarchy;
the method further comprises the steps of:
and after the data in the third hierarchy is merged into the second hierarchy, updating the second identifier, wherein the updated second identifier is different from the first identifier.
6. The method of claim 5, wherein the pointers comprise a read pointer and a write pointer;
judging whether the pointer is valid or not, comprising:
and when the first identifier and the second identifier are the same, determining that the read pointer and the write pointer are valid.
7. The method of claim 6, wherein the method further comprises:
and after resource positioning is carried out in the data storage system according to the target keyword, updating the first identifier, wherein the updated first identifier is the same as the second identifier.
8. The method according to claim 4 or 7, wherein when the pointer is determined to be valid, performing a data processing operation on the target data by the pointer, comprising at least one of the steps of:
reading the target data pointed by the read pointer;
and modifying the target data pointed by the write pointer.
9. The method according to claim 4 or 7, wherein reassigning the pointer according to the search result comprises:
if the target data is found, the memory address of the target data is assigned to the read pointer.
10. The method of claim 9, wherein performing a data processing operation on the target data via the reassigned pointer comprises:
and reading the target data pointed by the read pointer after reassignment.
11. The method according to claim 4 or 7, wherein reassigning the pointer according to the search result comprises:
if the target data is found, copying the target data to the first level;
and assigning the memory address of the target data in the first hierarchy to the write pointer.
12. The method of claim 11, wherein performing a data processing operation on the target data via the reassigned pointer comprises:
and modifying the target data pointed by the re-assigned write pointer.
13. The method according to claim 4 or 7, wherein reassigning the pointer according to the search result comprises:
if the target data is not found, creating an empty memory resource in the first level;
and assigning the memory address of the empty memory resource to the write pointer.
14. The method of claim 13, wherein performing a data processing operation on the target data via the reassigned pointer comprises:
and writing the target data in the empty memory resource pointed by the re-assigned write pointer.
15. A data processing apparatus for use in a LSM-Tree based data storage system, the apparatus comprising:
a receiving module, configured to receive a data processing request including a target keyword, where the data processing request is used to request to perform a data processing operation on target data corresponding to the target keyword;
the data processing module is used for executing data processing operation on the target data through a handle corresponding to the target keyword, wherein the handle comprises a pointer for pointing to the target data;
wherein the data processing module performs a data processing operation on the target data through a handle corresponding to the target keyword, including:
judging whether the pointer is valid or not;
performing a data processing operation on the target data by the pointer when the pointer is determined to be valid;
When the pointer is determined to be invalid, performing resource positioning in the data storage system according to the target keyword, and searching the target data;
reassigning the pointer according to the search result;
and executing data processing operation on the target data through the re-assigned pointer.
16. A data processing apparatus, comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor is configured to perform the data processing method of any of claims 1-14.
17. A non-transitory computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the data processing method of any of claims 1-14.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910198443.9A CN111694992B (en) | 2019-03-15 | 2019-03-15 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910198443.9A CN111694992B (en) | 2019-03-15 | 2019-03-15 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111694992A CN111694992A (en) | 2020-09-22 |
| CN111694992B true CN111694992B (en) | 2023-05-26 |
Family
ID=72475364
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910198443.9A Active CN111694992B (en) | 2019-03-15 | 2019-03-15 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111694992B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115934023A (en) * | 2021-08-31 | 2023-04-07 | 华为技术有限公司 | Data processing method, data processing device and related equipment |
| CN114398378B (en) * | 2022-03-25 | 2022-11-01 | 北京奥星贝斯科技有限公司 | Method and device for determining index cost |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105447059A (en) * | 2014-09-29 | 2016-03-30 | 华为技术有限公司 | Data processing method and device |
| CN105787093A (en) * | 2016-03-17 | 2016-07-20 | 清华大学 | Log file system construction method based on LSM-Tree structure |
| CN106844650A (en) * | 2017-01-20 | 2017-06-13 | 中国科学院计算技术研究所 | A kind of daily record merges the merging method and system of tree |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10592348B2 (en) * | 2016-06-17 | 2020-03-17 | Acronis International Gmbh | System and method for data deduplication using log-structured merge trees |
| US10303363B2 (en) * | 2016-10-19 | 2019-05-28 | Acronis International Gmbh | System and method for data storage using log-structured merge trees |
-
2019
- 2019-03-15 CN CN201910198443.9A patent/CN111694992B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105447059A (en) * | 2014-09-29 | 2016-03-30 | 华为技术有限公司 | Data processing method and device |
| CN105787093A (en) * | 2016-03-17 | 2016-07-20 | 清华大学 | Log file system construction method based on LSM-Tree structure |
| CN106844650A (en) * | 2017-01-20 | 2017-06-13 | 中国科学院计算技术研究所 | A kind of daily record merges the merging method and system of tree |
Non-Patent Citations (1)
| Title |
|---|
| 马文龙 ; 朱妤晴 ; 蒋德钧 ; 熊劲 ; 张立新 ; 孟潇 ; 包云岗 ; .Key-Value型NoSQL本地存储系统研究.计算机学报.2017,(08),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111694992A (en) | 2020-09-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102226257B1 (en) | Method and device for writing service data to a blockchain system | |
| CN109284321B (en) | Data loading method, device, computing equipment and computer readable storage medium | |
| CN110764748B (en) | Code calling method, device, terminal and storage medium | |
| CN111694992B (en) | Data processing method and device | |
| CN111241040B (en) | Information acquisition method and device, electronic equipment and computer storage medium | |
| CN113536174A (en) | Interface loading method, device and equipment | |
| CN111124480A (en) | Application package generation method and device, electronic equipment and storage medium | |
| JP7254925B2 (en) | Transliteration of data records for improved data matching | |
| CN110232156B (en) | Information recommendation method and device based on long text | |
| CN106156050B (en) | Data processing method and device | |
| CN111382179B (en) | Data processing method and device and electronic equipment | |
| CN111538667A (en) | Page testing method and device | |
| CN109409037B (en) | Method, device and equipment for generating data confusion rule | |
| CN107368281B (en) | Data processing method and device | |
| CN115934161A (en) | Code change influence analysis method, device and equipment | |
| CN112148746B (en) | Method, device, electronic device and storage medium for generating database table structure document | |
| CN114840427A (en) | Code testing and test case generating method and device | |
| CN110874322A (en) | Test method and test server for application program | |
| US9946755B2 (en) | Automated hybrid constraint database manager | |
| CN110245136B (en) | Data retrieval method, device, equipment and storage equipment | |
| CN109460255B (en) | Memory address query method and device | |
| CN111459398B (en) | Data processing method and device of distributed system | |
| CN111722785A (en) | Cache updating method and device | |
| CN112286572A (en) | Configuration method and device of business process | |
| CN110716946A (en) | Method and device for updating feature rule matching library, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |