CN117390071A

CN117390071A - Data connection method and device, storage medium and electronic device

Info

Publication number: CN117390071A
Application number: CN202311469932.6A
Authority: CN
Inventors: 刘大伟; 朱圣祥; 丁君
Original assignee: China Telecom Intelligent Network Technology Co ltd
Current assignee: China Telecom Intelligent Network Technology Co ltd
Priority date: 2023-11-03
Filing date: 2023-11-03
Publication date: 2024-01-12

Abstract

The application discloses a data connection method and device, a storage medium and an electronic device, wherein the method comprises the following steps: acquiring a first table and a second table, wherein the first table and the second table are provided with target fields; storing a set of field values corresponding to the target field in the first table into a first cuckoo filter; searching first data in a second table through the first cuckoo filter, and connecting the first data with data, which are matched with field values of target fields in the second data, wherein the first data comprise N pieces of data, which correspond to N field values stored in the first cuckoo filter, in the second table, the second data comprise N pieces of data, which correspond to N field values stored in the first cuckoo filter, in the first table, the field value set comprises N field values, and N is a positive integer greater than or equal to 1. By adopting the technical scheme, the problem of lower association connection efficiency when two database tables are associated and connected together is solved.

Description

Data connection method and device, storage medium and electronic device

Technical Field

The present invention relates to the field of data processing, and in particular, to a data connection method and apparatus, a storage medium, and an electronic device.

Background

With the high-speed development of the Internet, the data volume is also in a explosive growth state, and the problem of associated query optimization of the data table is also continuously explored and researched. In conventional database queries, it is often necessary to scan the two tables in full and compare the join key values piece by piece without using any optimization techniques, which is very inefficient.

In recent years, researchers have continuously optimized bloom filters to optimize the performance of the table hashjoin, but bloom filter designs have the problem of being unable to be deleted because each slot position of the bloom filter may be commonly occupied by a plurality of elements, and the original data needs to be loaded into the memory, so that the memory occupation is large.

In database queries, the filters commonly used are the count bloom filter, the D-left count bloom filter. Wherein, the counting bloom filter increases corresponding times on the memory pin due to the increase of the counter digit. The D-left counting bloom filter, while reducing memory usage by 50% compared to conventional counting bloom filters, requires knowledge of data volume problems in advance, and is only available in fixed scenarios and is not applicable to scenarios with dynamic data volume.

Aiming at the problem of low association connection efficiency when two database tables are associated and connected together in the related art, no effective solution is proposed at present.

Accordingly, there is a need for improvements in the related art to overcome the drawbacks of the related art.

Content of the application

The embodiment of the application provides a data connection method and device, a storage medium and an electronic device, which are used for at least solving the problem of low association connection efficiency when two database tables are associated and connected together.

According to one embodiment of the present application, there is provided a data connection method including: acquiring a first table and a second table, wherein the first table and the second table are provided with target fields; storing a set of field values corresponding to the target field in the first table into a first cuckoo filter; searching first data in the second table through the first cuckoo filter, and performing connection operation on the first data and data matched with field values of target fields in the second data, wherein the first data comprises N pieces of data corresponding to N field values stored in the first cuckoo filter in the second table, the second data comprises N pieces of data corresponding to N field values stored in the first cuckoo filter in the first table, the field value set comprises the N field values, and N is a positive integer greater than or equal to 1.

Optionally, storing the set of field values corresponding to the target field in the first table to a first cuckoo filter, including: performing hash operation on each field value in the field value set by using a first hash function to obtain a hash value set; and storing the hash value set into M barrels of the first cuckoo filter by using a consistent hash algorithm, wherein M barrel numbers corresponding to the M barrels are uniformly distributed on a hash ring corresponding to the first cuckoo filter, the M barrels and the M barrel numbers have a one-to-one correspondence, and M is a positive integer greater than or equal to N.

Optionally, storing the set of hash values into M buckets of the first cuckoo filter using a consistent hash algorithm, comprising: storing an ith hash value in the set of hash values into a bucket of the first cuckoo filter by: performing hash operation on the ith hash value by using a second hash function to obtain a first target number; and determining a first target barrel number with the smallest difference value with the first target barrel number from the M barrel numbers, and storing the ith hash value into the first target barrel corresponding to the first target barrel number.

Optionally, storing the ith hash value in a first target bucket corresponding to the first target bucket number includes: determining a target hash function corresponding to the first target bucket, and performing hash operation on the ith hash value by using the target hash function to obtain a slot number; storing the ith hash value into a slot corresponding to the slot number when no data exists in the slot corresponding to the slot number of the first target bucket; under the condition that a target hash value exists in a slot position corresponding to the first target bucket and the slot position number, determining a second target bucket according to the target hash value and the first target number, and storing the target hash value into the second target bucket; and storing the ith hash value into a slot corresponding to the slot number.

Optionally, after storing the set of field values corresponding to the target field in the first table in a first cuckoo filter, the method further includes: acquiring a Structured Query Language (SQL) statement before searching first data in the second table through the first cuckoo filter, wherein the SQL statement is used for deleting part of field values in the field value set; and determining a storage position of the partial field value in the first cuckoo filter, and deleting the partial field value of the storage position in the first cuckoo filter.

Optionally, after storing the set of field values corresponding to the target field in the first table in a first cuckoo filter, the method further includes: acquiring a Structured Query Language (SQL) statement before searching first data in the second table through the first cuckoo filter, wherein the SQL statement is used for deleting part of field values in the field value set; and constructing a second cuckoo filter, and storing the partial field value into the second cuckoo filter.

Optionally, searching the first data in the second table through the first cuckoo filter includes: searching first data in the second table through the first cuckoo filter and the second cuckoo filter; the first data comprises N pieces of data corresponding to N field values stored by the first cuckoo filter in the second table, and does not comprise partial data corresponding to partial field values stored by the second cuckoo filter in the second table; the second data includes N pieces of data in the first table corresponding to N field values stored by the first cuckoo filter, and does not include partial data in the first table corresponding to partial field values stored by the second cuckoo filter.

According to another embodiment of the present application, there is provided a data connection apparatus including: the system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is used for acquiring a first table and a second table, and the first table and the second table are provided with target fields; the storage module is used for storing a field value set corresponding to the target field in the first table into a first cuckoo filter; the connection module is configured to search first data in the second table through the first cuckoo filter, and perform a connection operation on the first data and data matched with a field value of a target field in the second data, where the first data includes N pieces of data corresponding to N field values stored in the first cuckoo filter in the second table, the second data includes N pieces of data corresponding to N field values stored in the first cuckoo filter in the first table, and the set of field values includes the N field values, and N is a positive integer greater than or equal to 1.

According to a further embodiment of the present application, there is also provided a computer readable storage medium having stored therein a computer program, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.

According to a further embodiment of the present application, there is also provided an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.

In the application, a first table and a second table are obtained, wherein the first table and the second table are provided with target fields; storing a set of field values corresponding to the target field in the first table into a first cuckoo filter; searching first data in the second table through the first cuckoo filter, and performing connection operation on the first data and data matched with field values of target fields in the second data, wherein the first data comprises N pieces of data corresponding to N field values stored in the first cuckoo filter in the second table, the second data comprises N pieces of data corresponding to N field values stored in the first cuckoo filter in the first table, the field value set comprises the N field values, and N is a positive integer greater than or equal to 1. By introducing a cuckoo filter to store and filter data in the process of associating and connecting two database tables, the memory access times during data query in the process of associating and connecting are reduced, the associating and connecting efficiency is improved, and the problem of lower associating and connecting efficiency when the two database tables are associated and connected together is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:

FIG. 1 is a block diagram of the hardware architecture of a computer device of a data connection method according to an embodiment of the present application;

FIG. 2 is a flow chart of a data connection method of an embodiment of the present application;

FIG. 3 is a flow chart of a hashjoin method of improving a cuckoo filter according to an embodiment of the present application;

FIG. 4 is a schematic flow diagram of a cuckoo filter selection tank according to an embodiment of the present application;

FIG. 5 is a consistent hash partitioning diagram according to an embodiment of the present application;

FIG. 6 is a schematic diagram of the locations of the buckets in a consistent hash ring in accordance with an embodiment of the present application;

FIG. 7 is a flow chart of a bucket for data storage according to an embodiment of the present application;

FIG. 8 is a flow diagram of data storage into various buckets according to an embodiment of the present application;

FIG. 9 is a schematic flow chart of a data storage capacity expansion bucket according to an embodiment of the present application;

fig. 10 is a block diagram of a data connection device according to an embodiment of the present application.

Detailed Description

In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms and "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

The method embodiments provided in the embodiments of the present application may be performed in a mobile terminal, a computer terminal or similar computing device. Taking the mobile terminal as an example, fig. 1 is a block diagram of a hardware structure of the mobile terminal of a data connection method according to an embodiment of the present application. As shown in fig. 1, the mobile terminal may include one or more (only one is shown in fig. 1) processors 102 (the processor 102 may include, but is not limited to, a microprocessor (Microprocessor Unit, abbreviated MPU) or programmable logic device (Programmable logic device, abbreviated PLD)) and a memory 104 for storing data, and in an exemplary embodiment, the mobile terminal may further include a transmission device 106 for communication functions and an input-output device 108. It will be appreciated by those skilled in the art that the structure shown in fig. 1 is merely illustrative and not limiting of the structure of the mobile terminal described above. For example, a mobile terminal may also include more or fewer components than shown in FIG. 1, or have a different configuration than the equivalent functions shown in FIG. 1 or more than the functions shown in FIG. 1.

The memory 104 may be used to store a computer program, for example, a software program of application software and a module, such as a computer program corresponding to a data connection method in the embodiment of the present application, and the processor 102 executes the computer program stored in the memory 104 to perform various functional applications and data processing, which corresponds to implementing the method described above. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor 102, which may be connected to the mobile terminal via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The transmission means 106 is arranged to receive or transmit data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, simply referred to as NIC) that can connect to other network devices through a base station to communicate with the internet. In one example, the transmission device 106 may be a Radio Frequency (RF) module, which is used to communicate with the internet wirelessly.

In order to solve the above-mentioned problem, a data connection method is provided in the present embodiment, and fig. 2 is a flowchart of a data connection method in the embodiment of the present application, as shown in fig. 2, the flowchart includes the following steps S202 to S206:

step S202: acquiring a first table and a second table, wherein the first table and the second table are provided with target fields;

the first table and the second table are two tables that need to be queried for association. The target field is a common field between the first table and the second table when the association query is made.

Step S204: storing a set of field values corresponding to the target field in the first table into a first cuckoo filter;

As an alternative example, the first table is a table with a smaller data amount, i.e. a table with a smaller number of rows, of the two tables, and the present application may determine the size tables of the first table and the second table based on the statistical information of the database and the query plan.

As an optional example, a hash operation may be performed on the field values in the field value set, and further, the hash value set corresponding to the field value set is stored in the first cuckoo filter.

As an alternative example, a consistent hashing algorithm may be used to store a set of field values in the first table that correspond to the target field into a first cuckoo filter.

It should be further noted that a Cuckoo Filter (Cuckoo Filter) is a probabilistic data structure for efficiently retrieving data, and by mapping elements to different positions using a hash function, it can be quickly determined whether an element exists in a collection. In addition, the cuckoo filter can preprocess the data before inquiring and judge whether each record meets the inquiring condition according to a certain probability model. By filtering out non-eligible records, the number of records that need to be compared can be greatly reduced.

Step S206: searching first data in the second table through the first cuckoo filter, and performing connection operation on the first data and data matched with field values of target fields in the second data, wherein the first data comprises N pieces of data corresponding to N field values stored in the first cuckoo filter in the second table, the second data comprises N pieces of data corresponding to N field values stored in the first cuckoo filter in the first table, the field value set comprises the N field values, and N is a positive integer greater than or equal to 1.

As an alternative example, the first data may be hashed with data matching the field value of the target field in the second data (hashjoin), which is a join algorithm commonly used in relational databases, to join the data of two tables according to a certain shared column. The connection mode maps the value of the connection column into the hash bucket through the hash function, so that efficient data matching is realized.

Through the steps S202-S206, the data is stored and filtered by introducing the cuckoo filter in the process of associating and connecting the two database tables, so that the memory access times during data query in the process of associating and connecting are reduced, the associating and connecting efficiency is improved, and the problem of lower associating and connecting efficiency when the two database tables are associated and connected together is solved.

In an exemplary embodiment, storing the set of field values corresponding to the target field in the first table to the first cuckoo filter may be implemented by the following steps S11-S12:

step S11: performing hash operation on each field value in the field value set by using a first hash function to obtain a hash value set;

the first hash function is a function that can map data of an arbitrary length to data of a fixed length. It generates a hash value of a fixed length from the input data by calculation, which is not further limited in the embodiments of the present application.

Step S12: and storing the hash value set into M barrels of the first cuckoo filter by using a consistent hash algorithm, wherein M barrel numbers corresponding to the M barrels are uniformly distributed on a hash ring corresponding to the first cuckoo filter, the M barrels and the M barrel numbers have a one-to-one correspondence, and M is a positive integer greater than or equal to N.

It should be noted that, the cuckoo filter is composed of M barrels, and is located on a hash ring corresponding to the cuckoo filter, as shown in fig. 4, which is a flow diagram of a cuckoo filter selecting barrel in an embodiment of the present application. The bucket number is the hash value corresponding to the bucket and is mapped to the value of a point on the hash ring.

That is, in this embodiment, hash operation is performed on field values of fields requiring hash connection in the first table to obtain a hash value set, and then the hash value set is stored into M buckets of the first cuckoo filter by using a consistent hash algorithm.

When elements are put into the cuckoo filter, the data are uniformly placed in each barrel of the filter by using a consistency hash algorithm, so that the repositioning times can be reduced, and invalid 'reject-insert' circulation operation is reduced.

As an alternative example, it is necessary to construct a consistent hash ring, calculate the hash value of m buckets in the cuckoo filter, and map the buckets to a point on the hash ring, so that the size of the cuckoo filter can be changed in response to the size of the data in the table, when the data size is smaller, the probability of repositioning is smaller, the same load of each bucket is basically maintained, when repositioning occurs, the load of the bucket is also changed in response to the occurrence, the shifting factor is used for adjustment, the cuckoo filter can be maintained to maintain the same load, and when the data size of the table is large, the number of buckets in the cuckoo filter can be properly enlarged, and the hash value can be calculated and placed on the ring as shown in fig. 9. Existing data will be remapped to new nodes based on their hash values. When the table data size is small, the responsive bucket needs to be removed to reduce the memory occupation, and the data related to the bucket can be remapped to the next node, so that a smooth migration process is realized, specifically, the new node can be determined by the following formula:

Wherein H' _i A hash value corresponding to a virtual bucket representing a bucket of an i-th cuckoo filter, i=1, 2,..m, m is the number of buckets; h _i Representing a hash value corresponding to the ith bucket;

when relocation occurs, the corresponding change bucket hash map is required to the consistent hash ring location:

wherein H is _M Representing a hash value corresponding to the barrel of the cuckoo filter after the position of the corresponding consistency hash ring is moved; h represents a hash value corresponding to the bucket of any cuckoo filter; h _N Representing the hash value corresponding to the surrounding nodes of the initial hash ring corresponding to the barrel of any cuckoo filter, and beta represents a shift factor for dynamically adjusting the position of the consistent hash ring corresponding to the barrel of the cuckoo filter.

It should be noted that, the element is balanced and loaded into m buckets by using the consistent hash algorithm, so that the relocation times can be effectively reduced.

In one exemplary embodiment, the ith hash value in the set of hash values may be stored into the bucket of the first cuckoo filter by the following steps S21-S22:

step S21: performing hash operation on the ith hash value by using a second hash function to obtain a first target number;

step S22: and determining a first target barrel number with the smallest difference value with the first target barrel number from the M barrel numbers, and storing the ith hash value into the first target barrel corresponding to the first target barrel number.

It should be noted that, each hash value has two alternative buckets, so that hash collision is avoided.

Since the numbers of the M buckets are preset on the hash ring, that is, there is no bucket directly corresponding to the first target number on the hash ring, it is necessary to select the first target bucket corresponding to the first target bucket number with the smallest difference between the first target numbers to store the ith hash value. Meanwhile, the bucket corresponding to the first target bucket number with the smallest difference value is selected, so that the element distribution in the hash table is more uniform, and the occurrence of hash collision is reduced. Therefore, the size of the cuckoo filter can be changed in response to the data amount in the table, when the data amount is smaller, the probability of repositioning is smaller, the same load of each barrel is basically kept, when repositioning occurs, the load of the barrel can also be changed in response to the change, the shifting factor is used for adjustment, and the cuckoo filter can be kept at the same load.

In an exemplary embodiment, the ith hash value may be stored in the first target bucket corresponding to the first target bucket number through the following steps S31-S33:

step S31: determining a target hash function corresponding to the first target bucket, and performing hash operation on the ith hash value by using the target hash function to obtain a slot number;

As an alternative example, assuming that the i-th hash value is x and the target hash function hash1 corresponding to the first target bucket, the slot number of the i-th hash value in the first target bucket is hash1 (x).

It should be noted that each barrel of the first cuckoo filter has n slots for storing fingerprint data of elements.

Step S32: storing the ith hash value into a slot corresponding to the slot number when no data exists in the slot corresponding to the slot number of the first target bucket;

as an alternative embodiment, if the hash1 (x) slot of the first target bucket has no value, the i-th hash value is directly stored into the hash1 (x) slot of the first target bucket.

Step S33: under the condition that a target hash value exists in a slot position corresponding to the first target bucket and the slot position number, determining a second target bucket according to the target hash value and the first target number, and storing the target hash value into the second target bucket; and storing the ith hash value into a slot corresponding to the slot number.

As an alternative example, if the hash1 (x) slot of the first target bucket has the target hash value y, an alternative bucket (assuming that the objective function of another bucket is hash 2) of the target hash value y is determined, and whether the hash2 (y) slot of the alternative bucket of y has data or not is determined, if no data exists, the target hash value y is stored to the hash2 (y) slot of the alternative bucket of y, and then the i-th hash value is stored to the hash1 (x) slot of the first target bucket.

It should be noted that if the hash2 (y) slot of the candidate bucket of y has the data z, determining the candidate bucket of z, storing the data z to the hash3 (z) slot of the candidate bucket of z, and then storing y to the hash2 (y) slot of the candidate bucket of y, and sequentially and circularly executing "reject-insert", wherein the hash3 is the target hash function corresponding to the candidate bucket of z.

As an alternative embodiment, determining the second target bucket from the target hash value and the first target number may be achieved by the following steps S1-S3:

step S1: determining the fingerprint of the target hash value, and calculating the fingerprint by using the second hash function to obtain an operation result;

step S2: performing exclusive OR operation on the first target number and the operation result to obtain a second target number;

as an alternative example, for the ith hash value, the numbers P1 and P2 of the two buckets corresponding to the ith hash value may be determined by the following formula:

P1＝hash(x)；P2＝P1^hash(fp)；fp＝fingerprint(x)；

wherein, hash is the second hash function, fp is the fingerprint of the target hash value, hash (fp) is the operation result, P1 is the first target number, and P2 is the second target number.

Step S3: and determining a second target barrel number with the smallest difference value with the second target barrel number from the M barrel numbers, and determining the second target barrel as a barrel corresponding to the second target barrel number.

In an exemplary embodiment, after storing the set of field values corresponding to the target field in the first table in the first cuckoo filter, the following steps S41 to S42 are further provided:

step S41: acquiring a Structured Query Language (SQL) statement before searching first data in the second table through the first cuckoo filter, wherein the SQL statement is used for deleting part of field values in the field value set;

step S42: and determining a storage position of the partial field value in the first cuckoo filter, and deleting the partial field value of the storage position in the first cuckoo filter.

That is, "not in" and "++in the where statement can be combined by SQL statement! And deleting keywords such as' and the like, and deleting the data which do not accord with the query result, so as to ensure the accuracy of the query data. The value to be deleted can be calculated through a hash function, and the field value positioned to the storage position in the cuckoo filter is deleted. If the field value does not exist in the position, the flow is skipped.

In an exemplary embodiment, after storing the set of field values corresponding to the target field in the first table in the first cuckoo filter, the following steps S51-S52 are further provided:

Step S51: acquiring a Structured Query Language (SQL) statement before searching first data in the second table through the first cuckoo filter, wherein the SQL statement is used for deleting part of field values in the field value set;

step S52: and constructing a second cuckoo filter, and storing the partial field value into the second cuckoo filter.

It should be noted that, because the number of buckets on the hash ring is limited, when the hash value is stored, a part of hash value is moved, so that the problem of false positive of the cuckoo filter may be caused by the deletion operation, and for the current situation of false positive, the data needs to be calculated to compare the memory size occupied by the error rate reduction with the memory size generated by the second cuckoo filter expansion, and the operation is performed in a manner of taking less occupied response memory. Most of the systems insensitive to data accuracy can tolerate the system, but the important data is not tolerated, at the moment, a double cuckoo filter is added, and the data to be deleted is loaded into a new cuckoo filter, so that the false positive problem can be effectively avoided.

In an exemplary embodiment, searching the first data in the second table through the first cuckoo filter may be implemented by the following step S61:

Searching first data in the second table through the first cuckoo filter and the second cuckoo filter; the first data comprises N pieces of data corresponding to N field values stored by the first cuckoo filter in the second table, and does not comprise partial data corresponding to partial field values stored by the second cuckoo filter in the second table; the second data includes N pieces of data in the first table corresponding to N field values stored by the first cuckoo filter, and does not include partial data in the first table corresponding to partial field values stored by the second cuckoo filter.

When the data is queried in the second table, the first cuckoo filter and the second cuckoo filter are applied to the data of the query large table, the data of the large data table are read, the data which are not matched with the first cuckoo filter and the second cuckoo filter are filtered, and then matched data records are searched.

It will be apparent that the embodiments described above are only some, but not all, of the embodiments of the present application. For better understanding of the above method, the following description will explain the above process with reference to the examples, but is not intended to limit the technical solutions of the embodiments of the present application, specifically:

The cuckoo filter can preprocess the data before inquiring and judge whether each record meets the inquiring condition according to a certain probability model. By filtering out non-eligible records, the number of records that need to be compared can be greatly reduced. Therefore, when hashjoin operation is performed, deletion operation can be supported, and memory overhead and CPU calculation amount are reduced.

Specifically, when using a cuckoo filter and hashjoin combination, the cuckoo filter is first constructed separately for both tables and stored in memory. Then, when the hashjoin operation is performed, firstly, the record is taken out from a table according to the connection key value, whether the record meets the query condition or not is judged through the cuckoo filter, the record is put into the cuckoo filter, the data is placed in the filter in an equalizing mode through a consistency hash algorithm, when the data size is too large, the requirement of dynamically increasing the barrel number of the cuckoo filter can be met through consistency hash, meanwhile, the repositioning times are reduced, and invalid 'reject-insert' circulation operation is reduced. Using cuckoo to support delete operations, solve "not in" and "≡in the sphere statement of sql! The problem = "the present application builds a dual cuckoo filter to support reliable deletion. And comparing records of the constructed cuckoo filter in another table of the associated query, wherein hit data is loaded with required data, otherwise, the method directly skips and does not carry out subsequent hashjoin operation. Therefore, the number of records to be compared can be greatly reduced, and the query efficiency is improved. In addition, through the combined use of the cuckoo filter and the hashjoin, the characteristic of reliable deletion can be met on the premise of ensuring the correctness of the query result, unnecessary hashjoin operation is reduced, and the query efficiency is remarkably improved.

Optionally, the present application provides a hashjoin method for improving a cuckoo filter, and fig. 3 is a flowchart of a hashjoin method for improving a cuckoo filter according to an embodiment of the present application, as shown in fig. 3, where the flowchart includes the following steps S1-S8:

step S1: receiving two tables needing to be related to query, firstly comparing and acquiring data of a small data table in the tables, and traversing each record;

before using hashjoin, step S1 first determines which table is the small table and which table is the large table. Typically, a small table refers to a table with a smaller number of rows, while a large table refers to a table with a larger number of rows. The present application determines the size table in the association table by statistics based on the database and the query plan.

Step S2: carrying out hash operation on fields needing hash connection in the small table to obtain a hash value;

step S3: constructing a cuckoo filter, wherein the filter consists of m barrels, each barrel is provided with n grooves, the grooves are used for storing fingerprint data of elements, m hash () functions are set, and the m barrels correspond to each other; two alternative barrels are available for each element, and fig. 4 is a schematic flow diagram of a cuckoo filter selection barrel according to an embodiment of the present application. Specifically, two candidate buckets are calculated by the partial-key Cuckoo hashing method: fp= fingerprint (x); p1=hash (x); p2=p1≡hash (fp);

P2 can be obtained by an exclusive OR operation of P1 and the element fingerprint. It should be noted that the complete x element value may not be required; meanwhile, the above formula can use the dual of exclusive OR, i.e. when knowing P2 and fp, P1 can also be calculated, i.e. P1=P2≡hash (fp).

When the element is stored in the barrel, if the hash1 (x) slot position of one barrel of the x element has no value, the element x is put in; if the hash1 (x) slot position of one bucket of x elements has a value y, putting x into the position of the hash1 (x), rejecting y, and putting y into the position of the hash2 (y) of the other bucket of y elements; ending if the hash2 (y) slot position has no value;

if the hash2 (y) slot position of the other bucket of the y element has a value z, y is put into the position of the hash2 (y), z is put into the position of the hash3 (z) of the other bucket of the z element, and the 'reject-insert' is sequentially and circularly executed;

step S4: inserting a data value of a cuckoo filter, and selecting one of m barrels from elements to be connected and circulated in the small table by using a consistency hash algorithm;

it should be noted that, before step S4, a consistent hash ring needs to be constructed, as shown in fig. 5, which is a schematic diagram of consistent hash partitioning in an embodiment of the present application, fig. 6 is a schematic diagram of positions of each bucket in a consistent hash ring in an embodiment of the present application, and one of m buckets is selected by using a consistent hash algorithm, so that elements can be uniformly loaded into the m buckets, as shown in fig. 7, which is a schematic diagram of a flow of selecting a bucket for data storage in an embodiment of the present application, and fig. 8 is a schematic diagram of a flow of storing data into each bucket in an embodiment of the present application. Compared with the traditional algorithm of randomly selecting barrels, the method can effectively reduce repositioning times, reduce invalid 'reject-insert' circulation operation, and dynamically adjust the size of the cuckoo filter according to data quantity change through a consistency hash algorithm.

Through the consistency hash ring, the hash value of m barrels in the cuckoo filter is calculated, the barrels are mapped to a point on the hash ring, so that the size of the cuckoo filter can be changed in response to the size of data in a table, when the data size is smaller, the probability of repositioning is smaller, the same load of each barrel is basically kept, when repositioning is generated, the load of the barrels can also be changed in response, the shifting factor is used for adjustment, the same load can be kept for the cuckoo filter, and when the data in the table is large, the number of the barrels in the cuckoo filter can be properly enlarged, the hash value of the cuckoo filter is calculated, and the cuckoo filter is placed on the ring. The existing data is remapped to a new node according to its hash value, as shown in fig. 9, which is a flow chart of a data storage capacity expansion bucket according to an embodiment of the present application. When the amount of table data is small, the responsive bucket needs to be removed to reduce memory consumption, and the data associated with the bucket is remapped to its next node.

Alternatively, the formula may be:and a smooth migration process is realized.

when relocation occurs, the corresponding position of the change bucket hash mapping to the consistent hash ring is required, and can be realized by the following formula:

It should be noted that, the consistent hash algorithm loads elements into m buckets in a balanced manner, so that relocation times can be effectively reduced.

As can be derived from step S3, each element can be uniformly distributed to the extent that when the i+1st insertion of the data element is performed, the occurrence of relocation occurs because the position of hash1 (x) has a value y when the i+1st insertion is performed, and the position of hash2 (y) also has a value when y is found to be a new position, and the probability of occurrence of relocation when the i+1st insertion is as follows:

wherein P is ₁ (A _i+1 ) Representing the probability of repositioning for the i+1th insertion, m is the cuckoo filter bucket size, n is the number of elements that each bucket can hold.

Assuming that relocation has occurred, when the number of times relocation occurs is 1, the probability can be calculated as follows:

from this it can be deduced that a relocation has occurred, and when the number of relocation occurrences is x, the probability can be calculated as follows:

the expected function of the number of relocations that can occur is thus:

when the (i+1) th element is inserted, mapping is performed through hash1 (x) and hash2 (x) into two buckets, wherein the bucket corresponding to hash1 already contains a elements, and the bucket corresponding to hash2 already contains b elements, which can be represented by the following manner:

in the case where the number of elements in the bucket is 0< a < b < n, the probabilities assigned to the two buckets are equal when the i+1th element is inserted using a random algorithm, but when the i+1th element is inserted using a consistent hash algorithm, the data insertion probability is tilted toward the bucket corresponding to hash 1.

The difference in probability of relocation generated by comparing the random selection and the consistency hash is as follows:

from the above, it can be seen that the use of consistent hash to load the elements into the barrel of the cuckoo filter reduces the probability of repositioning as compared to the prior randomly selected insertion of elements into the barrel of the cuckoo filter. And the more significant the efficiency of the improved repositioning as the number of insertions increases.

Step S5: selecting a slot position in the bucket by utilizing a hash (x) function for the element x, and processing a repositioning problem;

step S6: check "not in" and "+.! Keywords such as= "and deleting the data which do not accord with the query result from the filter; and calculating a value to be filtered through the original hash () function, positioning to the position of a specific placed element in the candidate barrel, deleting if the position has the value, and skipping if the data element value does not exist in the cuckoo filter.

Here, the operation of deletion may cause a cuckoo filter false positive problem, which may be found by the formula:

where f is the length of the element and n is the amount of the element that each bucket can hold.

If the setting tolerates a certain error rate epsilon,

average the number of bits occupied by each element asa represents a load factor.

When certain data accuracy is required, the error rate epsilon is reduced, and the probability of false positive of the cuckoo filter is required to be reduced, so that the memory usage is also increased sharply.

Therefore, the relevant judgment is used for comparing the memory size occupied by the computing element for reducing the error rate with the memory size generated by the expanding second cuckoo filter for the scene which can tolerate a certain error rate, and the operation is performed in a mode of smaller response memory occupation.

For the current situation of false positives, most systems insensitive to data accuracy can tolerate the situation, but for scenes involving user detailed information, finance and the like, a double cuckoo filter is added, and data to be deleted is loaded into a new cuckoo filter.

Step S7: the cuckoo filter 1 generated in the step S4 and the cuckoo filter 2 generated in the step S6 are applied to data of a query large table, a large amount of data table data is read, the data of the unmatched cuckoo filter 1 is filtered, if a hash value exists in the filter 1, a method for tolerating error rate or a mode for creating a double cuckoo filter can be selected according to a strategy for deleting in the S6 method in a real scene, and a matched record is further searched in the hash table 1.

Step S8: and performing connection operation on the two records with the matched connection fields, and returning the result.

Through the application, the following technical effects can be achieved:

1. solve the "not in" and "≡in the sphere statement of sql! = "etc., solves the false positive problem of the original cuckoo filter, ensures the accuracy of data.

2. When elements are placed into the cuckoo filter, the data are placed in each barrel of the filter in a balanced mode by using a consistency hash algorithm, the repositioning times are reduced, and invalid rejection-insertion cyclic operation is reduced.

3. The cuckoo filter is used for reducing the memory access times during inquiry, and the efficiency is higher.

From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus the necessary general hardware platform, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method of the embodiments of the present application.

In this embodiment, a data connection device is further provided, and the device is used to implement the foregoing embodiments and preferred embodiments, and will not be described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the devices described in the following embodiments are preferably implemented in software, implementations in hardware, or a combination of software and hardware, are also possible and contemplated.

Fig. 10 is a block diagram of a data connection device according to an embodiment of the present application, and as shown, the device includes:

an obtaining module 1002, configured to obtain a first table and a second table, where the first table and the second table each have a target field therein;

a storage module 1004, configured to store a set of field values corresponding to the target field in the first table into a first cuckoo filter;

and a connection module 1006, configured to search, by using the first cuckoo filter, first data in the second table, and perform a connection operation on the first data and data that matches a field value of a target field in the second data, where the first data includes N pieces of data in the second table that corresponds to N field values stored by the first cuckoo filter, the second data includes N pieces of data in the first table that corresponds to N field values stored by the first cuckoo filter, and the set of field values includes the N field values, and N is a positive integer greater than or equal to 1.

By means of the device, the cuckoo filter is introduced into the process of associating and connecting the two database tables to store and filter data, so that the memory access times during data query in the process of associating and connecting are reduced, the associating and connecting efficiency is improved, and the problem that the associating and connecting efficiency is lower when the two database tables are associated and connected together is solved.

In an exemplary embodiment, the storage module 1004 is further configured to perform a hash operation on each field value in the set of field values using a first hash function to obtain a set of hash values; and storing the hash value set into M barrels of the first cuckoo filter by using a consistent hash algorithm, wherein M barrel numbers corresponding to the M barrels are uniformly distributed on a hash ring corresponding to the first cuckoo filter, the M barrels and the M barrel numbers have a one-to-one correspondence, and M is a positive integer greater than or equal to N.

In an exemplary embodiment, the storage module 1004 is further configured to store an ith hash value in the set of hash values into the bucket of the first cuckoo filter by: performing hash operation on the ith hash value by using a second hash function to obtain a first target number; and determining a first target barrel number with the smallest difference value with the first target barrel number from the M barrel numbers, and storing the ith hash value into the first target barrel corresponding to the first target barrel number.

In an exemplary embodiment, the storage module 1004 is further configured to determine a target hash function corresponding to the first target bucket, and perform a hash operation on the i-th hash value using the target hash function to obtain a slot number; storing the ith hash value into a slot corresponding to the slot number when no data exists in the slot corresponding to the slot number of the first target bucket; under the condition that a target hash value exists in a slot position corresponding to the first target bucket and the slot position number, determining a second target bucket according to the target hash value and the first target number, and storing the target hash value into the second target bucket; and storing the ith hash value into a slot corresponding to the slot number.

In an exemplary embodiment, the apparatus further includes a first processing module, configured to obtain a structured query language SQL statement after storing, in a first cuckoo filter, a set of field values corresponding to the target field in the first table, and before searching, by the first cuckoo filter, for first data in the second table, where the SQL statement is configured to delete a portion of the field values in the set of field values; and determining a storage position of the partial field value in the first cuckoo filter, and deleting the partial field value of the storage position in the first cuckoo filter.

In an exemplary embodiment, the apparatus further includes a second processing module, configured to obtain a structured query language SQL statement after storing, in a first cuckoo filter, a set of field values corresponding to the target field in the first table, and before searching, by the first cuckoo filter, for first data in the second table, where the SQL statement is configured to delete a portion of the field values in the set of field values; and constructing a second cuckoo filter, and storing the partial field value into the second cuckoo filter.

In an exemplary embodiment, the connection module 1006 is further configured to search the second table for first data through the first cuckoo filter and the second cuckoo filter; the first data comprises N pieces of data corresponding to N field values stored by the first cuckoo filter in the second table, and does not comprise partial data corresponding to partial field values stored by the second cuckoo filter in the second table; the second data includes N pieces of data in the first table corresponding to N field values stored by the first cuckoo filter, and does not include partial data in the first table corresponding to partial field values stored by the second cuckoo filter.

Embodiments of the present application also provide a computer readable storage medium having a computer program stored therein, wherein the computer program is arranged to perform the steps of any of the method embodiments described above when run.

In one exemplary embodiment, the computer readable storage medium may include, but is not limited to: a usb disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing a computer program.

Specific examples in this embodiment may refer to the examples described in the foregoing embodiments and the exemplary implementation, and this embodiment is not described herein.

Embodiments of the present application also provide an electronic device comprising a memory having a computer program stored therein and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.

In an exemplary embodiment, the electronic device may further include a transmission device connected to the processor, and an input/output device connected to the processor.

It will be appreciated by those skilled in the art that the modules or steps of the application described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than that shown or described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps of them may be fabricated into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application and are intended to be comprehended within the scope of the present application.

Claims

1. A method of data connection, comprising:

acquiring a first table and a second table, wherein the first table and the second table are provided with target fields;

storing a set of field values corresponding to the target field in the first table into a first cuckoo filter;

searching first data in the second table through the first cuckoo filter, and performing connection operation on the first data and data matched with field values of target fields in the second data, wherein the first data comprises N pieces of data corresponding to N field values stored in the first cuckoo filter in the second table, the second data comprises N pieces of data corresponding to N field values stored in the first cuckoo filter in the first table, the field value set comprises the N field values, and N is a positive integer greater than or equal to 1.

2. The method of claim 1, wherein storing the set of field values in the first table corresponding to the target field to a first cuckoo filter comprises:

performing hash operation on each field value in the field value set by using a first hash function to obtain a hash value set;

and storing the hash value set into M barrels of the first cuckoo filter by using a consistent hash algorithm, wherein M barrel numbers corresponding to the M barrels are uniformly distributed on a hash ring corresponding to the first cuckoo filter, the M barrels and the M barrel numbers have a one-to-one correspondence, and M is a positive integer greater than or equal to N.

3. The method of claim 2, wherein storing the set of hash values into M buckets of the first cuckoo filter using a consistent hash algorithm comprises:

storing an ith hash value in the set of hash values into a bucket of the first cuckoo filter by:

performing hash operation on the ith hash value by using a second hash function to obtain a first target number;

and determining a first target barrel number with the smallest difference value with the first target barrel number from the M barrel numbers, and storing the ith hash value into the first target barrel corresponding to the first target barrel number.

4. The method of claim 3, wherein storing the ith hash value in the first destination bucket corresponding to the first destination bucket number comprises:

determining a target hash function corresponding to the first target bucket, and performing hash operation on the ith hash value by using the target hash function to obtain a slot number;

storing the ith hash value into a slot corresponding to the slot number when no data exists in the slot corresponding to the slot number of the first target bucket;

under the condition that a target hash value exists in a slot position corresponding to the first target bucket and the slot position number, determining a second target bucket according to the target hash value and the first target number, and storing the target hash value into the second target bucket; and storing the ith hash value into a slot corresponding to the slot number.

5. The method of claim 1, wherein after storing the set of field values in the first table corresponding to the target field in a first cuckoo filter, the method further comprises:

acquiring a Structured Query Language (SQL) statement before searching first data in the second table through the first cuckoo filter, wherein the SQL statement is used for deleting part of field values in the field value set;

And determining a storage position of the partial field value in the first cuckoo filter, and deleting the partial field value of the storage position in the first cuckoo filter.

6. The method of claim 1, wherein after storing the set of field values in the first table corresponding to the target field in a first cuckoo filter, the method further comprises:

and constructing a second cuckoo filter, and storing the partial field value into the second cuckoo filter.

7. The method of claim 6, wherein looking up first data in the second table through the first cuckoo filter comprises:

searching first data in the second table through the first cuckoo filter and the second cuckoo filter;

the first data comprises N pieces of data corresponding to N field values stored by the first cuckoo filter in the second table, and does not comprise partial data corresponding to partial field values stored by the second cuckoo filter in the second table; the second data includes N pieces of data in the first table corresponding to N field values stored by the first cuckoo filter, and does not include partial data in the first table corresponding to partial field values stored by the second cuckoo filter.

8. A data connection device, comprising:

the system comprises an acquisition module, a storage module and a storage module, wherein the acquisition module is used for acquiring a first table and a second table, and the first table and the second table are provided with target fields;

the storage module is used for storing a field value set corresponding to the target field in the first table into a first cuckoo filter;

the connection module is configured to search first data in the second table through the first cuckoo filter, and perform a connection operation on the first data and data matched with a field value of a target field in the second data, where the first data includes N pieces of data corresponding to N field values stored in the first cuckoo filter in the second table, the second data includes N pieces of data corresponding to N field values stored in the first cuckoo filter in the first table, and the set of field values includes the N field values, and N is a positive integer greater than or equal to 1.

9. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored program, wherein the program when run performs the method of any of the preceding claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method according to any of the claims 1 to 7 by means of the computer program.