CN117215500A - Method and system for breaking through LMDBKey length - Google Patents
Method and system for breaking through LMDBKey length Download PDFInfo
- Publication number
- CN117215500A CN117215500A CN202311481850.3A CN202311481850A CN117215500A CN 117215500 A CN117215500 A CN 117215500A CN 202311481850 A CN202311481850 A CN 202311481850A CN 117215500 A CN117215500 A CN 117215500A
- Authority
- CN
- China
- Prior art keywords
- data
- user
- lmdb
- user data
- shortening
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000004904 shortening Methods 0.000 claims abstract description 96
- 238000006243 chemical reaction Methods 0.000 claims description 61
- 238000004590 computer program Methods 0.000 claims description 10
- 230000009466 transformation Effects 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Abstract
The application belongs to the technical field of data processing, and discloses a method and a system for breaking through the length of an LMDB Key, wherein the method comprises the steps of obtaining user data to be stored in the LMDB; judging whether shortening processing is needed to be carried out on the user data based on the user data; if shortening processing is needed to be carried out on the user data, shortening the user data by using a preset shortening algorithm to obtain shortened user data stored in the LMDB; judging whether the user history data in the LMDB is matched with the user data or not; if the user history data is matched with the user data, shortening the user history data into shortened user data stored in the LMDB by utilizing a preset conflict resolution algorithm, and combining the user history data and conflict part data in the user data into combined data stored in the LMDB; compared with the prior art, the method shortens the length of the original Key by shortening the algorithm, thereby breaking through the length of the Key in the LMDB.
Description
Technical Field
The application belongs to the technical field of data processing, and particularly relates to a method and a system for breaking through the length of an LMDB Key.
Background
LMDB (Lightning Memory-Mapped Database Manager ), which is a btree-based database management library; LMDB is loosely modeled in berkeley db API (Key-value structured embedded database engine) to achieve data storage simplification. The whole LMDB database is disclosed in the memory mapping, all data acquisition directly returns data from the mapped memory, and malloc (dynamic memory allocation) or memcpy (memory copy function) does not appear during the data acquisition; therefore, the LMDB database is very simple, does not need a page cache layer, and has very high performance and memory efficiency; just as LMDB is widely used because it has small, flexible, embedded features, etc.
However, LMDB has some drawbacks, in particular, the Key length, which default (default) cannot exceed 512 bytes (i.e. the maximum is 511 bytes, which is determined when the LMDB database is compiled), and in some specific scenarios, the limitation of the Key length may cause it to fail to meet the application requirements.
Disclosure of Invention
The application provides a method and a system for breaking through the length of an LMDB Key, which are used for breaking through the length of the Key in the LMDB.
In a first aspect, the object of the application is achieved by the following technical scheme:
a method of breaking through LMDB Key length, comprising:
acquiring user data to be stored in an LMDB;
judging whether shortening processing is needed to be carried out on the user data or not based on the user data;
if the user data needs to be shortened, shortening the user data by using a preset shortening algorithm to obtain shortened user data stored in an LMDB;
judging whether user history data in the LMDB is matched with the user data or not;
and if the user history data is matched with the user data, shortening the user history data into the shortened user data stored in the LMDB by utilizing a preset conflict resolution algorithm, and combining the user history data and conflict part data in the user data into combined data stored in the LMDB.
According to the technical scheme, user data to be stored in the LMDB are firstly obtained, whether shortening treatment is needed to be carried out on the user data is judged according to the user data and a preset treatment limit value, if yes, the user data is shortened according to a shortening algorithm and then stored in the LMDB to be used as shortening user data; then judging whether a plurality of user history data stored in the LMDB are matched with the user data or not, if the plurality of user history data are matched with the user data, shortening the user history data into shortened user data stored in the LMDB according to a conflict resolution algorithm, and combining the user history data and conflict part data in the user data into combined data stored in the LMDB, thereby reducing the occupation of the storage space of the LMDB; compared with the prior art, the method shortens the length of the user data by shortening the algorithm, thereby breaking through the length of Key in the LMDB.
The application is further provided with: the determining whether to shorten the user data based on the user data and a preset processing limit value includes:
acquiring the original Key length according to the original Key in the user data;
judging whether shortening processing is needed to be carried out on the user data according to whether the original Key length is larger than a preset processing limit value or not;
if the original Key length is not greater than a preset processing limit value, shortening processing is not needed for the user data, and the user data is rewritten into an LMDB;
if the original Key length is greater than a preset processing limit value, shortening the user data is needed.
According to the technical scheme, when the original Key length in the user data is smaller than the preset processing limit value, the original Key length of the user data can be directly stored in the storage space of each limited byte length of the LMDB, and shortening processing of the user data is not needed; when the original Key length in the user data is greater than a preset processing limit value, shortening the user data; compared with the prior art, the method fully considers two conditions of the long Key and the short Key, performs different data processing aiming at the two conditions, and has higher universality.
The application is further provided with: the shortening the user data by using a preset shortening algorithm, and obtaining the shortened user data stored in the LMDB includes:
extracting an original Key in the user data;
dividing the original Key into prefix data and residual data according to a preset prefix length, wherein the residual data is data to be converted in the original Key;
shortening and converting the residual data by using a preset HASH algorithm to obtain converted data;
and combining the prefix data and the conversion data to obtain shortened user data stored in the LMDB.
According to the technical scheme, the shortening algorithm divides the original Key into the prefix data and the residual data according to the preset prefix length, then shortens and converts the residual data through the preset HASH algorithm to obtain converted data, and finally combines the prefix data and the converted data to obtain shortened user data stored in the LMDB; compared with the prior art, the method shortens the residual data of the original Key through the HASH algorithm, and breaks through the length of the Key in the LMDB.
The application is further provided with: the step of shortening and converting the residual data by using a preset HASH algorithm to obtain converted data comprises the following steps:
if the length of the residual data exceeds the maximum conversion length of a preset HASH algorithm, dividing the residual data into first residual data and second residual data;
the length of the first residual data is equal to the maximum conversion length;
shortening and converting the first residual data by using a preset HASH algorithm to obtain converted data;
and storing the second remaining data to the combined data of the LMDB.
Through the technical scheme, when the length of the residual data exceeds the preset maximum conversion length of the HASH algorithm, the residual data is divided into the first residual data and the second residual data, and the length of the first residual data is equal to the maximum conversion length of the HASH algorithm so as to shorten the residual data to the greatest extent; the first residual data is obtained corresponding conversion data through a HASH algorithm, and the second residual data is stored in the combined data of the LMDB.
The application is further provided with: the determining whether the user history data in the LMDB matches the user data includes:
prefix history data and conversion history data of user history data are obtained;
if the prefix history data and the prefix data are repeated and the conversion history data and the conversion data are repeated, the user history data and the user data are matched;
if the prefix history data is different from the prefix data or the conversion history data is different from the conversion data, the user history data is not matched with the user data.
According to the technical scheme, when prefix historical data and prefix data of user data are repeated and conversion data of conversion historical data and conversion data of user data are repeated, the user historical data and the user data are indicated to be matched; otherwise, the user history data and the user data are not matched, and a related flow of conflict resolution is not required to be executed.
The application is further provided with: the shortening the user history data into the shortened user data stored in the LMDB by using a preset conflict resolution algorithm, and combining the user history data and conflict part data in the user data into combined data stored in the LMDB includes:
covering the prefix history data with the prefix data and covering the conversion history data with the conversion data by using a preset conflict resolution algorithm;
and combining the user history data and conflict part data in the user data into combined data stored in an LMDB by using a preset conflict resolution algorithm.
Through the technical scheme, when the user history data and the user data are matched, the prefix history data is firstly overwritten into the prefix data by utilizing a conflict resolution algorithm, and then the conversion history data is overwritten into the conversion data, so that the possibility that the Key in the repeated user data occupies excessive LMDB storage space is reduced, and the utilization rate of the LMDB storage space is improved; and combining the user history data and conflict part data in the user data into combined data stored in the LMDB, thereby realizing optimal storage of a plurality of user data in the LMDB.
The application is further provided with: after the user history data does not match the user data if the prefix history data is different from the prefix data or the conversion history data is different from the conversion data, the method further comprises:
and if the user history data is not matched with the user data, directly writing the user history data into the combined data of the LMDB.
Through the technical scheme, when the user history data and the user data are not matched, the related process of conflict resolution is not required to be executed, and the user history data are directly written into the combined data of the LMDB so as to meet the storage requirement of a plurality of user data in the LMDB.
In a second aspect, the object of the present application is achieved by the following technical solutions:
a system for breaking through the length of an LMDB Key, which is applied to the above method for breaking through the length of an LMDB Key, the method comprises:
the acquisition module is used for acquiring the user data to be stored in the LMDB;
the first judging module is used for judging whether shortening processing is needed to be carried out on the user data or not based on the user data;
the shortening processing module is used for shortening the user data by utilizing a preset shortening algorithm if shortening processing is needed to be carried out on the user data, so as to obtain shortened user data stored in the LMDB;
the second judging module is used for judging whether the user history data in the LMDB is matched with the user data or not;
and the conflict resolution module is used for shortening the user history data into the shortened user data stored in the LMDB by utilizing a preset conflict resolution algorithm if the user history data is matched with the user data, and combining the user history data and conflict part data in the user data into combined data stored in the LMDB.
In a third aspect, the object of the present application is achieved by the following technical solutions:
a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program being a method of achieving a breakthrough in LMDB Key length as described above.
In a fourth aspect, the object of the present application is achieved by the following technical solutions:
a computer readable storage medium storing a computer program which when executed by a processor implements a method of breaking through an LMDB Key length as described above.
In summary, the present application includes at least one of the following beneficial technical effects:
1. compared with the prior art, the method shortens the original Key through the shortening algorithm, so that the length of the Key in the LMDB is broken through.
2. Compared with the prior art, the method fully considers two conditions of the long Key and the short Key, performs different data processing aiming at the two conditions, and has higher universality.
3. Compared with the prior art, the method shortens the original Key through the HASH algorithm, and breaks through the length of the Key in the LMDB.
Drawings
FIG. 1 is a flow chart of a method of breaking through the LMDB Key length in accordance with an embodiment of the present application;
fig. 2 is a flowchart of step S20 in the method for breaking through the LMDB Key length according to the first embodiment of the present application;
FIG. 3 is a schematic diagram of a shortening algorithm in accordance with a first embodiment of the present application;
fig. 4 is a flowchart of step S30 in the method for breaking through the LMDB Key length according to the first embodiment of the present application;
fig. 5 is a flowchart of step S33 in the method for breaking through the LMDB Key length according to the first embodiment of the present application;
fig. 6 is a flowchart of step S40 in the method of breaking through the LMDB Key length in the first embodiment of the present application;
FIG. 7 is a schematic diagram of a conflict resolution algorithm in accordance with a first embodiment of the present application;
fig. 8 is a flowchart of step S50 in the method for breaking through the LMDB Key length according to the first embodiment of the present application;
FIG. 9 is a schematic block diagram of a system for breaking through the LMDB Key length in accordance with the second embodiment of the present application;
fig. 10 is a schematic diagram of a computer device in a third embodiment of the application.
Detailed Description
The technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments, and all other embodiments obtained by those skilled in the art without making creative efforts based on the embodiments of the present application are included in the protection scope of the present application.
The embodiment of the application provides a method and a system for breaking through the length of an LMDB Key, which are used for breaking through the length of the Key in the LMDB.
Example 1
As shown in fig. 1, a method for breaking through the length of an LMDB Key according to an embodiment of the present application includes:
s10: user data to be stored to the LMDB is acquired.
In this embodiment, the user data includes data that needs to be stored by various users, such as text data, image data, and image data; each user data contains keys and values that are adapted to LMDB storage.
Specifically, key and Value in the user data are acquired.
S20: based on the user data, it is determined whether shortening of the user data is required.
Specifically, judging whether the Key length in the user data exceeds a preset processing limit value, if so, shortening the user data; if not, the shortening process is not performed on the user data.
As shown in fig. 2 and 3, step S20 includes:
s21: acquiring the original Key length according to the original Key in the user data;
s22: judging whether shortening processing is needed to be carried out on the user data according to whether the original Key length is larger than a preset processing limit value or not;
s23: if the original Key length is not greater than the preset processing limit value, shortening processing is not needed for the user data, and the user data is overwritten into the LMDB;
s24: if the original Key length is greater than a preset processing limit value, shortening the user data is needed.
In this embodiment, the preset processing limit value is represented by MaxKPL in the program document; wherein, the actual limit length of the MaxKPL is set according to the actual situation; the following is a table of the relationship between Key and Value in the user data in step S21;
list one
It should be noted that table one is merely illustrative and does not represent actual user data in the method of the present application.
Specifically, when the original Key length in the user data is smaller than MaxKPL, the original Key length of the user data can be directly stored into a storage space with a limited byte length in the LMDB, and the user data is directly overwritten into the LMDB; when the original Key length of the user data is larger than MaxKPL, shortening the user data is needed.
Compared with the prior art, the method considers the situation that the original Key length is larger than MaxKPL or smaller than MaxKPL, and executes corresponding data processing flow aiming at the two situations so as to improve the utilization rate of the storage space in the LMDB, and has higher universality.
S30: if the user data needs to be shortened, shortening the user data by using a preset shortening algorithm to obtain shortened user data stored in the LMDB.
Specifically, the Key length of the user data is shortened according to a shortening algorithm and then stored in the LMDB, so that the length of the Key in the LMDB is broken through.
As shown in fig. 3 and 4, step S30 includes:
s31: extracting an original Key in user data;
s32: dividing the original Key into prefix data and residual data according to the preset prefix length, wherein the residual data is the data to be converted in the original Key;
s33: shortening and converting the residual data by using a preset HASH algorithm to obtain converted data;
s34: and combining the prefix data and the conversion data to obtain shortened user data stored in the LMDB.
In this embodiment, the stored Key Prefix data of the original Key Prefix data stored in the LMDB is expressed by Key Prefix in the handler.
Specifically, extracting an original Key in user data, and dividing the original Key into prefix data and residual data according to a preset prefix length, wherein the residual data is data, which is required to be converted by the HASH algorithm, of the original Key; copying and storing prefix data of the original Key into an LMDB as a prefix of the stored Key; after the prefix data is copied and stored, the shortening algorithm calls a HASH algorithm, carries out HASH operation on the residual data of the original Key, converts the residual data into a HASH value with a fixed length (namely, converted data), and additionally stores the HASH value to the prefix of the stored Key so as to realize the combination of the prefix data and the converted data, thereby obtaining the shortened user data stored to the LMDB (namely, the stored Key).
Compared with the prior art, the method shortens the residual data of the original Key through the HASH algorithm, and breaks through the length of the Key in the LMDB.
As shown in fig. 3 and 5, step S33 includes:
s331: if the length of the residual data exceeds the maximum conversion length of a preset HASH algorithm, dividing the residual data into first residual data and second residual data;
s332: the length of the first residual data is equal to the maximum conversion length;
s333: shortening and converting the first residual data by using a preset HASH algorithm to obtain converted data;
s334: the second remaining data is stored to the combined data of the LMDB.
Specifically, when the length of the residual data exceeds the preset maximum conversion length of the HASH algorithm, dividing the residual data into first residual data and second residual data, wherein the length of the first residual data is equal to the maximum conversion length of the HASH algorithm, so as to shorten the residual data to the greatest extent and store the long Key in the LMDB as much as possible; the first residual data obtains a corresponding HASH value through a HASH algorithm, and the second residual data is stored in the combined data of the LMDB.
S40: and judging whether the user history data in the LMDB is matched with the user data.
In this embodiment, the user history data is also data stored in the LMDB by a shortening algorithm; wherein the user history data includes history shortened user data; and judging whether the user history data in the LMDB is matched with the user data or not, namely judging whether the history shortening user data in the LMDB and the shortening user data are repeated or not.
Specifically, it is determined whether the history shortened user data and the shortened user data within the LMDB are repeated.
As shown in fig. 6 and 7, step S40 includes:
s41: prefix history data and conversion history data of user history data are obtained;
s42: if the prefix history data and the prefix data are repeated and the conversion history data and the conversion data are repeated, the user history data and the user data are matched;
s43: if the prefix history data is different from the prefix data or the conversion history data is different from the conversion data, the user history data is not matched with the user data.
In this embodiment, a specific implementation method for determining whether a plurality of user history data in the LMDB and a newly stored LMDB match is as follows:
watch II
Watch III
As shown in table one and table two, the prefix history data and prefix data are repeated and the conversion history data and conversion data are repeated, which means that the user history data and the user data are matched, i.e. the user data is shortened and the history is shortened.
Table four
As shown in table two and table four, if the prefix history data is different from the prefix data or the conversion history data is different from the conversion data, the user history data is not matched with the user data, i.e. the user data is shortened and the history shortened user data is not repeated.
S50: if the user history data is matched with the user data, shortening the user history data into shortened user data stored in the LMDB by utilizing a preset conflict resolution algorithm, and combining the user history data and conflict part data in the user data into combined data stored in the LMDB.
In this embodiment, the conflict part data includes Value of the user data, second remaining data of the user data, value of the user history data, and second remaining data of the user history data.
Specifically, as shown in fig. 7, when the user history data matches the user data, the user history data is shortened to shortened user data stored to the LMDB using a conflict resolution algorithm, and Value of the user history data, vaule in the user data, second remaining data of the user data, and second remaining data of the user history data are combined to combined data stored to the LMDB, thereby achieving optimal storage of a plurality of user data within the LMDB.
As shown in fig. 7 and 8, step S50 includes:
s51: utilizing a preset conflict resolution algorithm to cover prefix historical data with prefix data and cover conversion historical data with conversion data;
s52: and combining the user history data and conflict part data in the user data into combined data stored in the LMDB by using a preset conflict resolution algorithm.
Specifically, when the history shortens the user data and shortens the repetition of the user data, the prefix history data is overwritten into the prefix data and the conversion history data is overwritten into the conversion data by utilizing a conflict resolution algorithm, so that the possibility that keys in the repeated user data occupy excessive LMDB storage space is reduced, and the utilization rate of the LMDB storage space is improved.
Wherein, after step S40, the method of the present application further comprises:
if the user history data is not matched with the user data, the user history data is directly written into the combined data of the LMDB.
Specifically, when the history shortening user data and the shortening user data are not repeated, the user history data are directly written into the combined data of the LMDB without executing a related flow of conflict resolution, so that the storage requirements of a plurality of user data in the LMDB are met, and the operation amount is reduced.
Example two
As shown in fig. 9, an embodiment of the present application discloses a system for breaking through the length of an LMDB Key, which is used for executing the method for breaking through the length of an LMDB Key, and the system for breaking through the length of an LMDB Key corresponds to the method for breaking through the length of an LMDB Key in the above embodiment.
The embodiment of the application discloses a system for breaking through the length of an LMDB Key, which comprises the following steps:
an acquisition module 10 for acquiring user data to be stored to the LMDB;
a first judging module 20, configured to judge whether shortening processing is required for the user data based on the user data;
a shortening processing module 30, configured to shorten the user data by using a preset shortening algorithm if shortening processing is required for the user data, so as to obtain shortened user data stored in the LMDB;
a second judging module 40, configured to judge whether the user history data in the LMDB matches the user data;
and a conflict resolution module 50, configured to shorten the user history data to the shortened user data stored in the LMDB by using a preset conflict resolution algorithm if the user history data is matched with the user data, and combine the user history data and the conflict part data in the user data to form combined data stored in the LMDB.
The first judging module 20 includes:
a length obtaining sub-module, configured to obtain an original Key length according to the original Key in the user data;
a shortening judging sub-module, configured to judge whether shortening processing is required for the user data according to whether the original Key length is greater than a preset processing limit value;
the overlay writing sub-module is used for enabling the user data to be overlaid and written into the LMDB without shortening the user data if the original Key length is not larger than a preset processing limit value;
and the shortening instruction submodule is used for shortening the user data if the original Key length is larger than a preset processing limit value.
Wherein the shortening processing module 30 includes:
the data extraction sub-module is used for extracting the original Key in the user data;
the first dividing sub-module is used for dividing the original Key into prefix data and residual data according to a preset prefix length, wherein the residual data is data to be converted in the original Key;
the first conversion sub-module is used for shortening and converting the residual data by utilizing a preset HASH algorithm to obtain converted data;
and the data combination sub-module is used for combining the prefix data and the conversion data to obtain shortened user data stored in the LMDB.
Wherein the shortening processing module 30 includes:
the second dividing sub-module is used for dividing the residual data into first residual data and second residual data if the length of the residual data exceeds the maximum conversion length of a preset HASH algorithm;
a transition length defining sub-module, the length of the first remaining data being equal to the maximum transition length;
the second conversion sub-module is used for shortening and converting the first residual data by utilizing a preset HASH algorithm to obtain converted data;
and the first storage sub-module is used for storing the second residual data into the combined data of the LMDB.
The second judging module 40 includes:
the data acquisition sub-module is used for acquiring prefix historical data and conversion historical data of the user historical data;
a first match determination sub-module that matches the user history data with the user data if the prefix history data and the prefix data are repeated and the conversion history data and the conversion data are repeated;
and the second matching judging sub-module is used for enabling the user history data to be not matched with the user data if the prefix history data is different from the prefix data or the conversion history data is different from the conversion data.
Wherein the conflict resolution module 50 comprises:
the overlay writing sub-module is used for overlaying the prefix historical data on the prefix data and overlaying the conversion historical data on the conversion data by utilizing a preset conflict resolution algorithm;
and the second storage sub-module is used for combining the user history data and the conflict part data in the user data into combined data stored in the LMDB by utilizing a preset conflict resolution algorithm.
The system for breaking through the length of the LMDB Key according to the present embodiment can achieve each step of the foregoing embodiment due to the function of each module and the logic connection between each module, so that the same technical effects as those of the foregoing embodiment can be achieved, and the principle analysis can refer to the related description of the steps of the foregoing method for breaking through the length of the LMDB Key, which is not repeated herein.
For a specific limitation of a system for breaking through the length of the LMDB Key, reference may be made to the limitation of a method for breaking through the length of the LMDB Key, which is not described herein; each module in the system for breaking through the LMDB Key length can be fully or partially realized by software, hardware and combination thereof; each of the above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may invoke and execute the operations corresponding to each of the above modules.
Example III
As shown in fig. 10, in the present embodiment, a computer apparatus includes a memory, a processor, a nonvolatile storage medium, and a computer program stored in the nonvolatile storage medium and executable on the processor, the processor implementing the following steps when executing the computer program:
acquiring user data to be stored in an LMDB;
judging whether shortening processing is needed to be carried out on the user data based on the user data;
if shortening processing is needed to be carried out on the user data, shortening the user data by using a preset shortening algorithm to obtain shortened user data stored in the LMDB;
judging whether the user history data in the LMDB is matched with the user data or not;
if the user history data is matched with the user data, shortening the user history data into shortened user data stored in the LMDB by utilizing a preset conflict resolution algorithm, and combining the user history data and conflict part data in the user data into combined data stored in the LMDB.
In this embodiment, there is provided a computer-readable storage medium storing a computer program which when executed performs the steps of:
acquiring user data to be stored in an LMDB;
judging whether shortening processing is needed to be carried out on the user data based on the user data;
if shortening processing is needed to be carried out on the user data, shortening the user data by using a preset shortening algorithm to obtain shortened user data stored in the LMDB;
judging whether the user history data in the LMDB is matched with the user data or not;
if the user history data is matched with the user data, shortening the user history data into shortened user data stored in the LMDB by utilizing a preset conflict resolution algorithm, and combining the user history data and conflict part data in the user data into combined data stored in the LMDB.
Those skilled in the art will appreciate that implementing all or part of the above described embodiment methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of each of the above described embodiments. Any reference to memory, storage, database, or other medium used in each of the embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a number of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink), DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of each functional unit and module is illustrated, and in practical application, the above-described functional allocation may be performed by different functional units and modules, that is, the internal result of the apparatus is divided into different functional units or modules, so as to perform all or part of the above-described functions.
The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that; the technical scheme described in each embodiment can be modified or part of the characteristics can be replaced equivalently; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of each embodiment of the present application, and are intended to be included in the scope of the present application.
Claims (10)
1. A method of breaking through the length of an LMDB Key comprising:
acquiring user data to be stored in an LMDB;
judging whether shortening processing is needed to be carried out on the user data or not based on the user data;
if the user data needs to be shortened, shortening the user data by using a preset shortening algorithm to obtain shortened user data stored in an LMDB;
judging whether user history data in the LMDB is matched with the user data or not;
and if the user history data is matched with the user data, shortening the user history data into the shortened user data stored in the LMDB by utilizing a preset conflict resolution algorithm, and combining the user history data and conflict part data in the user data into combined data stored in the LMDB.
2. The method for breaking through the LMDB Key length according to claim 1, wherein the determining whether shortening the user data is required based on the user data comprises:
acquiring the original Key length according to the original Key in the user data;
judging whether shortening processing is needed to be carried out on the user data according to whether the original Key length is larger than a preset processing limit value or not;
if the original Key length is not greater than a preset processing limit value, shortening processing is not needed for the user data, and the user data is rewritten into an LMDB;
if the original Key length is greater than a preset processing limit value, shortening the user data is needed.
3. The method for breaking through the Key length of the LMDB according to claim 1, wherein the shortening the user data by using a preset shortening algorithm, obtaining the shortened user data stored in the LMDB comprises:
extracting an original Key in the user data;
dividing the original Key into prefix data and residual data according to a preset prefix length, wherein the residual data is data to be converted in the original Key;
shortening and converting the residual data by using a preset HASH algorithm to obtain converted data;
and combining the prefix data and the conversion data to obtain shortened user data stored in the LMDB.
4. A method for breaking through the length of LMDB Key according to claim 3, wherein said performing shortened transformation on said remaining data using a preset HASH algorithm to obtain transformed data comprises:
if the length of the residual data exceeds the maximum conversion length of a preset HASH algorithm, dividing the residual data into first residual data and second residual data;
the length of the first residual data is equal to the maximum conversion length;
shortening and converting the first residual data by using a preset HASH algorithm to obtain converted data;
and storing the second remaining data to the combined data of the LMDB.
5. A method of breaking through an LMDB Key length according to claim 3, wherein said determining whether user history data within the LMDB matches said user data comprises:
prefix history data and conversion history data of user history data are obtained;
if the prefix history data and the prefix data are repeated and the conversion history data and the conversion data are repeated, the user history data and the user data are matched;
if the prefix history data is different from the prefix data or the conversion history data is different from the conversion data, the user history data is not matched with the user data.
6. A method of breaking through an LMDB Key length according to claim 5, wherein the shortening the user history data into the shortened user data stored to the LMDB using a predetermined conflict resolution algorithm, and combining the user history data and the conflict part data in the user data into combined data stored to the LMDB comprises:
covering the prefix history data with the prefix data and covering the conversion history data with the conversion data by using a preset conflict resolution algorithm;
and combining the user history data and conflict part data in the user data into combined data stored in an LMDB by using a preset conflict resolution algorithm.
7. A method of breaking through an LMDB Key length according to claim 5, wherein after the user history data does not match the user data if the prefix history data is different from the prefix data or the conversion history data is different from the conversion data, the method further comprises:
and if the user history data is not matched with the user data, directly writing the user history data into the combined data of the LMDB.
8. A system for breaking through the length of an LMDB Key, characterized in that it is applied to a method for breaking through the length of an LMDB Key as claimed in any of claims 1 to 7, said system comprising:
the acquisition module is used for acquiring the user data to be stored in the LMDB;
the first judging module is used for judging whether shortening processing is needed to be carried out on the user data or not based on the user data;
the shortening processing module is used for shortening the user data by utilizing a preset shortening algorithm if shortening processing is needed to be carried out on the user data, so as to obtain shortened user data stored in the LMDB;
the second judging module is used for judging whether the user history data in the LMDB is matched with the user data or not;
and the conflict resolution module is used for shortening the user history data into the shortened user data stored in the LMDB by utilizing a preset conflict resolution algorithm if the user history data is matched with the user data, and combining the user history data and conflict part data in the user data into combined data stored in the LMDB.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing a method of breaking through the length of an LMDB Key as claimed in any of claims 1 to 7 when the computer program is executed.
10. A computer readable storage medium storing a computer program which when executed by a processor implements a method of breaking through LMDB Key length as claimed in any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311481850.3A CN117215500B (en) | 2023-11-09 | 2023-11-09 | Method and system for breaking through LMDB Key length |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311481850.3A CN117215500B (en) | 2023-11-09 | 2023-11-09 | Method and system for breaking through LMDB Key length |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117215500A true CN117215500A (en) | 2023-12-12 |
| CN117215500B CN117215500B (en) | 2024-01-23 |
Family
ID=89051520
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311481850.3A Active CN117215500B (en) | 2023-11-09 | 2023-11-09 | Method and system for breaking through LMDB Key length |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117215500B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5778255A (en) * | 1995-10-10 | 1998-07-07 | International Business Machines Corporation | Method and system in a data processing system for decompressing multiple compressed bytes in a single machine cycle |
| US20080228933A1 (en) * | 2007-03-12 | 2008-09-18 | Robert Plamondon | Systems and methods for identifying long matches of data in a compression history |
| JP2012169831A (en) * | 2011-02-14 | 2012-09-06 | Fujitsu Ltd | Traffic data monitoring system and method for matching data between servers |
| CN104298687A (en) * | 2013-07-18 | 2015-01-21 | 阿里巴巴集团控股有限公司 | Hash partitioning management method and device |
| CN108460044A (en) * | 2017-02-20 | 2018-08-28 | 阿里巴巴集团控股有限公司 | The treating method and apparatus of data |
| CN112988763A (en) * | 2021-05-14 | 2021-06-18 | 北京顺实科技有限公司 | Kv database storage and query method using partial hash Key as index |
| CN114610650A (en) * | 2022-03-11 | 2022-06-10 | Oppo广东移动通信有限公司 | Memory compression method and device, storage medium and electronic equipment |
| CN116150207A (en) * | 2023-02-13 | 2023-05-23 | 中国银联股份有限公司 | Data caching method and device, electronic equipment and storage medium |
-
2023
- 2023-11-09 CN CN202311481850.3A patent/CN117215500B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5778255A (en) * | 1995-10-10 | 1998-07-07 | International Business Machines Corporation | Method and system in a data processing system for decompressing multiple compressed bytes in a single machine cycle |
| US20080228933A1 (en) * | 2007-03-12 | 2008-09-18 | Robert Plamondon | Systems and methods for identifying long matches of data in a compression history |
| JP2012169831A (en) * | 2011-02-14 | 2012-09-06 | Fujitsu Ltd | Traffic data monitoring system and method for matching data between servers |
| CN104298687A (en) * | 2013-07-18 | 2015-01-21 | 阿里巴巴集团控股有限公司 | Hash partitioning management method and device |
| CN108460044A (en) * | 2017-02-20 | 2018-08-28 | 阿里巴巴集团控股有限公司 | The treating method and apparatus of data |
| CN112988763A (en) * | 2021-05-14 | 2021-06-18 | 北京顺实科技有限公司 | Kv database storage and query method using partial hash Key as index |
| CN114610650A (en) * | 2022-03-11 | 2022-06-10 | Oppo广东移动通信有限公司 | Memory compression method and device, storage medium and electronic equipment |
| CN116150207A (en) * | 2023-02-13 | 2023-05-23 | 中国银联股份有限公司 | Data caching method and device, electronic equipment and storage medium |
Non-Patent Citations (1)
| Title |
|---|
| 王忠效: "基于字符串匹配的通用数据压缩算法", 计算机应用, no. 01 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117215500B (en) | 2024-01-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9003381B2 (en) | Context-specific optimized code | |
| CN109597571B (en) | Data storage method, data reading method, data storage device, data reading device and computer equipment | |
| CN110555041A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN112579602A (en) | Multi-version data storage method and device, computer equipment and storage medium | |
| CN112306503A (en) | Front-end resource packaging method and device, computer equipment and storage medium | |
| CN112559529A (en) | Data storage method and device, computer equipment and storage medium | |
| CN108037932B (en) | SPI-NAND configuration file acquisition method and device | |
| CN117215500B (en) | Method and system for breaking through LMDB Key length | |
| CN107229655B (en) | Updating method, device and system of page static file | |
| CN110795197A (en) | Method and device for internationalizing enumerated classes, computer equipment and storage medium | |
| CN112632080B (en) | Data storage method, device and equipment based on block chain | |
| CN112783599A (en) | Virtual machine live migration method and device, terminal equipment and computer storage medium | |
| CN110647336A (en) | Gray scale distribution method and device, computer equipment and storage medium | |
| CN112783866A (en) | Data reading method and device, computer equipment and storage medium | |
| CN111736869A (en) | Version updating method of server-side interface and calling method of server-side interface | |
| CN109063044B (en) | Parameter insertion method and device for database table, terminal equipment and storage medium | |
| CN112860287A (en) | System program upgrading method and device, computer equipment and storage medium | |
| JP2016207172A (en) | Data processing apparatus and program | |
| CN110704437A (en) | Method, device, equipment and storage medium for modifying database query statement | |
| CN110381487A (en) | Bluetooth connecting method and device | |
| US10795875B2 (en) | Data storing method using multi-version based data structure | |
| CN113934752B (en) | Spark component-based data processing method, device, equipment and medium | |
| WO2021129048A1 (en) | Method, device, and apparatus for writing file data and storage medium | |
| CN110718285B (en) | Data archiving method, system, device and storage medium | |
| CN110162507B (en) | Snapshot overwriting method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |