WO2016029664A1

WO2016029664A1 - Two-dimensional filter generation method, query method and device

Info

Publication number: WO2016029664A1
Application number: PCT/CN2015/072915
Authority: WO
Inventors: 张延松; 陈红; 李翠平; 孙东旺
Original assignee: 华为技术有限公司
Priority date: 2014-08-28
Filing date: 2015-02-12
Publication date: 2016-03-03
Also published as: CN104317795A; EP3179382A4; JP2017526081A; RU2017109924A; US20170170968A1; EP3179382A1; RU2017109924A3

Abstract

A two-dimensional filter generation method, query method and device; the two-dimensional filter comprises a two-dimensional matrix capable of being linked to a plurality of key-value element groups, thus improving the flexibility of the filter; and in addition, when querying whether a key-value element belongs to the key-value elements comprised by the plurality of key-value element groups, only querying based on the two-dimensional filter without generating bloom filters respectively corresponding to each key-value element group, and without respectively querying based on a plurality of bloom filters, thus addressing the current defect of low query efficiency.

Description

Method, device and device for generating two-dimensional filter

Technical field

The present invention relates to the technical field of element query matching, and in particular to a method and device for generating and querying a two-dimensional filter.

Background technique

When designing computer software, it is often necessary to determine if an element is in a collection. For example, in word processing software, it is necessary to check whether an English word is spelled correctly (that is, to determine whether it is in a known dictionary); for example, in a URL (Universal Resource Locator) filtering software, judge one Whether the URL is in the filter list, and so on. The most straightforward way is to store all the elements in the collection in the computer. When a new element is encountered, it can be directly compared with the elements in the collection. To improve the speed of the lookup, a hash table is usually used to store the collection. A hash table is a data structure that quickly maps its storage location based on the key value of the element. This mapping function is what we usually call a hash function. The structure of the hash table is shown in Figure 1A. The elements in the set first get their hash position through the hash function, and then record the element in the hash list of the location. In Fig. 1A, assuming that the hash function is HASH, and A1, A2, ..., A8 are elements in the set, it can be seen from the figure that HASH(A1)=HASH(A2)=H1, HASH(A3)=HASH( A4)=H2, HASH(A5)=HASH(A6)=H3, HASH(A7)=HASH(A8)=H4.

The advantage of a hash table is that it can quickly and accurately determine whether an element is in a collection. The disadvantage is that it requires a large storage space. In order to save storage space, the one-dimensional Bloom filter was proposed by Barton Bron in 1970. The principle is as follows: A one-dimensional Bloom filter consists of k independent hash functions h1, h2, ..., Hk is composed of a bit vector of length m, where each hash function has a range of {0, 1, ..., m-1}, and because one byte has 8 bits, the bit vector The actual occupied memory space is m/8 bytes, and all bits of the bit vector are initialized to 0. Set S={s1, s2, ..., sn}, calculate a hash sequence for each element in set S with k hash functions (h1(s), h2(s), ..., hk(s)), and then set the corresponding hash sequence bit in the bit vector to 1, then the Bloom filter is said to be loaded with the data element set S, Or the Bloom filter represents a set of data elements S. For example, if h1(s1)=5, the 6th bit of the bit vector is set to 1, and h2(s1)=10, then the 11th bit of the bit vector is set to 1, until hk(s1)=n-1, Setting the nth bit of the bit vector to 1 means that the data element s1 is loaded in the Bloom filter. When each data element in the set S is loaded into the Bloom filter, the Bloom filter is called The data element set S. When querying whether a data element is in the set S, use the same k hash functions to calculate a hash sequence for the data element. If each bit on the bit vector corresponding to the hash sequence is 1, it is considered The data element belongs to S, otherwise it does not belong to S. Compared to the full storage of data, the use of Bloom filters saves storage space, and the use of Bloom filters never misses any element belonging to the collection.

The following is a simple description of the Bloom filter in conjunction with the example of a spam email address.

Assuming there are 100 million spam email addresses, first create a bit vector of 1.6 billion bits, which is a two-billion-byte vector, and then initialize all of the 1.6 billion bits to zero. For each known spam email address, eight hash values (f1, f2, ..., f8) are generated using eight different hash functions (F1, F2, ..., F8). The position of the bit vector corresponding to the hash value is set to 1. After processing the hundreds of millions of spam email addresses, a Bloom filter for these spam email addresses is generated. As shown in FIG. 1B, an example map represented by a Bloom filter for a garbage email address is used. For the garbage email address XXX@163.com, eight different hash functions (F1, F2, ..., F8) are used. The eight hash values (f1, f2, ..., f8) are generated, and the positions of the bit vectors corresponding to the eight hash values are all set to 1, and the spam address is loaded in the Bloom filter.

The Bloom filter generated above is for a group of key-value elements, that is, a Bloom filter can only be linked with a key-value element group, and several key-value element groups are required to generate several Bronze. The filter, therefore, has the drawback that the Bloom filter is less flexible.

Summary of the invention

Embodiments of the present invention provide a method and a device for generating and querying a two-dimensional filter, which are used to improve Bloom filter flexibility.

The specific technical solutions provided by the embodiments of the present invention are as follows:

In a first aspect, a method of querying a key value element is provided, comprising:

Determining, from the hash function set, a hash function sub-set corresponding to each group of key value element groups;

For each key value element in each group of key value element groups, a hash value is calculated according to the hash function sub-set corresponding to the key value element group to which the key value element belongs, and the calculated hash value is in the two-dimensional matrix. The element corresponding to the location is set as the second preset identifier;

Determining, according to the key value element to be queried, a hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, and calculating the key value to be queried according to the corresponding hash function sub-set The hash value of the element;

Obtaining an element corresponding to a position of the hash value of the to-be-queried key value element in the two-dimensional matrix;

Determining that the key value element to be queried belongs to two-dimensional when the acquired element is a second preset identifier corresponding to a position of the hash value of the to-be-queried key element in the two-dimensional matrix. A collection of key-valued elements represented by the filter.

With reference to the first aspect, in the first possible implementation manner of the first aspect, the hash function included in the hash function sub-set corresponding to any two different key element groups is different; or

The hash function included in the hash function sub-set corresponding to any two different key element groups is the same, and the hash function is arranged differently.

With reference to the first aspect, or the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, the calculating the hash value of the key value element to be queried includes:

Calculating, according to the hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, calculating a first hash value for the first sub-key value element of the key value element to be queried;

And calculating, according to the hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, a second hash value is calculated for the second sub-key value element of the key value element to be queried.

In conjunction with the second possible implementation of the first aspect, the third possible implementation in the first aspect In the current mode, the element corresponding to the position of the hash value of the to-be-queried key element in the two-dimensional matrix is obtained, which specifically includes:

Obtaining, in the two-dimensional matrix, an element having the first hash value as a row and the second hash value as a column; or acquiring the second hash value in the two-dimensional matrix, The first hash value is an element of the column.

In a second aspect, a method for generating a two-dimensional filter is provided, including:

Establishing a two-dimensional matrix comprising at least two row vectors and at least two column vectors;

Determining a set of hash functions, wherein each hash function in the set of hash functions corresponds to at least one group of key value elements, and any one of the hash function sets corresponds to at least one key The first sub-key value element of any key value element in the value element group is hashed to obtain a first hash value, and the second sub-key value element of the any key value element is hashed to obtain a second hash value, The first hash value is a positive integer that is less than or equal to the length of the row vector, and the second hash value is a positive integer that is less than or equal to the length of the column vector;

A two-dimensional filter including the two-dimensional matrix and the set of hash functions is generated.

With reference to the second aspect, in a first possible implementation manner of the second aspect, the length of the row vector and the length of the column vector are both greater than or equal to

Wherein, Sr is the number of all the key element elements included in all the key element group; or Sr is the number of key element elements obtained by filtering all the key value elements included in the all key element group.

With reference to the second aspect, or the first possible implementation manner of the second aspect, in a second possible implementation manner of the second aspect, the first sub-key value element includes the a key value element consisting of all odd bits, the second sub-key value element including a key value element consisting of all even bits of the binary key element when represented by a binary representation; or

The first sub-key value element includes a key value element consisting of a first bit to a Kth bit when the any of the key value elements is expressed in binary, and the second sub-key value element includes when the any of the key value elements is expressed in binary The key value element consisting of the K+1th to the Nth bits, N is the number of bits when any of the key elements is expressed in binary, 1≤K≤N, and K is a positive integer.

With reference to the second aspect, or the first to the second possible implementation manners of the second aspect, in a third possible implementation manner of the second aspect, the method further includes:

An element determined by any one of the two-dimensional matrix and any one of the columns of vectors is initialized to a first preset identifier.

In a third aspect, an apparatus for querying a key value element is provided, comprising:

a determining unit, configured to determine, from the hash function set, a hash function sub-collection corresponding to each group of key value element groups;

a setting unit, configured to calculate a hash value according to a hash function sub-set corresponding to the key value element group of the key value element group for each key value element group in each group of key value element groups, and calculate the hash value in The element corresponding to the position in the two-dimensional matrix is set as the second preset identifier;

a calculation unit, configured to determine, according to the key value element to be queried, a hash function subset corresponding to the key element group to which the key value element to be queried belongs, and calculate the hash function according to the corresponding hash function subset The hash value of the key element to be queried;

An acquiring unit, configured to acquire an element corresponding to a location of the hash value of the group of key element elements to be queried in the two-dimensional matrix;

a query unit, configured to determine, when the acquired element is the second preset identifier corresponding to a location of the hash value of the to-be-queried key element in the two-dimensional matrix, The key-value element belongs to a collection of key-valued elements represented by a two-dimensional filter.

With reference to the third aspect, in a first possible implementation manner of the third aspect, the determining unit includes a hash function included in a hash function sub-set corresponding to any two different key element groups respectively determined by the determining unit Different; or

The hash function included in the hash function sub-set corresponding to any two different key element group determined by the determining unit is the same, and the hash function is arranged differently.

With reference to the third aspect, or the first possible implementation manner of the third aspect, in a second possible implementation manner of the third aspect, the calculating unit is specifically configured to:

In conjunction with the second possible implementation of the third aspect, in a third possible implementation manner of the third aspect, the acquiring unit is specifically configured to:

In a fourth aspect, a device for generating a two-dimensional filter includes:

Establishing a unit for establishing a two-dimensional matrix comprising at least two row vectors and at least two column vectors;

a determining unit, configured to determine a hash function set, wherein each hash function in the hash function set corresponds to at least one key value element group, and any one of the hash function sets The first sub-key value element of any one of the corresponding at least one key value element group is hashed to obtain a first hash value, and the second sub-key value element of the any of the key value elements is hashed to obtain a a second hash value, the first hash value being a positive integer less than or equal to the length of the row vector, and the second hash value being a positive integer less than or equal to the length of the column vector;

And a generating unit, configured to generate a two-dimensional filter including the two-dimensional matrix and the hash function set.

With reference to the fourth aspect, in a first possible implementation manner of the fourth aspect, the two-dimensional matrix generated by the establishing unit includes a length of a row vector and a length of the column vector that are greater than or equal to

With reference to the fourth aspect, or the first possible implementation manner of the fourth aspect, in the second possible implementation manner of the fourth aspect, the first sub-key value element obtained by the determining unit includes any one of the key value elements a key value element consisting of all odd bits when represented by a binary, and the second sub-key value element obtained by the determining unit includes a key value element consisting of all even bits of the binary element when the binary element is represented; or

The first sub-key value element obtained by the determining unit includes a key value element composed of the first to the Kth bits when the any key value element is expressed in binary, and the second sub-key value element obtained by the determining unit includes the A key value element consisting of a K+1th bit to an Nth bit when any of the key value elements is expressed in binary, and N is a bit number when the any of the key value elements is expressed in binary, 1≤K≤N, and K is a positive integer.

With reference to the fourth aspect, or the first to second possible implementation manners of the fourth aspect, in a third possible implementation manner of the fourth aspect, An element determined by any one row vector and any one column vector is initialized to a first preset identifier.

In the embodiment of the present invention, a two-dimensional filter includes a two-dimensional matrix, and the two-dimensional matrix can be linked with a plurality of key element groups, thereby improving the flexibility of the filter.

DRAWINGS

1A is a schematic structural diagram of a hash table in the prior art;

FIG. 1B is a diagram showing an example of a garbage email address in the prior art using a filter; FIG.

2A is a flowchart of generating a two-dimensional filter in an embodiment of the present invention;

2B is a schematic diagram of a two-dimensional matrix in an embodiment of the present invention;

3 is a flowchart of querying a key value element in an embodiment of the present invention;

4 is an embodiment of generating a two-dimensional filter and querying a key value element according to an embodiment of the present invention;

FIG. 5 is a schematic diagram showing the functional structure of an apparatus for generating a two-dimensional filter according to an embodiment of the present invention; FIG.

FIG. 6 is a schematic diagram of a physical structure of an apparatus for generating a two-dimensional filter according to an embodiment of the present invention; FIG.

FIG. 7 is a schematic diagram showing the functional structure of an apparatus for querying a key value element according to an embodiment of the present invention; FIG.

FIG. 8 is a schematic diagram showing the physical structure of an apparatus for querying a key value element according to an embodiment of the present invention.

detailed description

The technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. It is a partial embodiment of the invention, and not all of the embodiments. Based on the embodiments of the present invention, those of ordinary skill in the art obtain the following without creative efforts. All other embodiments obtained are within the scope of the invention.

Additionally, the terms "system" and "network" are used interchangeably herein. The term "and/or" in this context is merely an association describing the associated object, indicating that there may be three relationships, for example, A and / or B, which may indicate that A exists separately, and both A and B exist, respectively. B these three situations. In addition, the letter "/" in this article generally indicates that the contextual object is an "or" relationship.

The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings, and the preferred embodiments of the present invention are intended to illustrate and explain the invention, and not to limit the invention, and The embodiments in the application and the features in the embodiments may be combined with each other.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Embodiment 1

Referring to FIG. 2A, in the embodiment of the present invention, a detailed process of generating a two-dimensional filter is as follows:

Step 200: Establish a two-dimensional matrix including at least two row vectors and at least two column vectors;

Step 210: Determine a hash function set, wherein each hash function in the hash function set corresponds to at least one key value element group, and any one of the hash function pairs in the hash function set corresponds to at least one key value The first sub-key value element of any key element in the element group is hashed to obtain a first hash value, and the second sub-key value element of any key element element in the corresponding key value element group is hashed to obtain a second a hash value, the first hash value is a positive integer less than or equal to the length of the row vector, and the second hash value is a positive integer less than or equal to the length of the column vector;

Step 220: Generate a two-dimensional filter including a two-dimensional matrix and a hash function set.

In the embodiment of the present invention, the two-dimensional matrix is as shown in FIG. 2B.

In the embodiment of the present invention, the number of the storage data units of the two-dimensional matrix is the product of the number of row vectors and the number of column vectors. As shown in FIG. 2B, the number of row vectors of the two-dimensional matrix is 9, and the number of column vectors is 9. Then, the number of stored data units of the two-dimensional matrix is 81.

In the embodiment of the present invention, if the length of the row vector and the column vector of the established two-dimensional matrix are smaller than

If the key value elements in the key element group are loaded into the filter, the probability that the different key value elements are loaded into the same position is high, which affects the accuracy of the query. Therefore, in order to improve the accuracy of the query, the embodiment of the present invention The length of the row vector and the column vector of the established two-dimensional matrix are greater than

Where Sr is the number of all key-value elements included in all key-value element groups; or, Sr is the number of key-valued elements that are filtered by the query condition for all key-value elements included in all key-value element groups.

However, the larger the length of the row vector and the column vector, the larger the required storage space. Therefore, in the embodiment of the present invention, in order to improve the utilization of the storage space, the row of the two-dimensional matrix is established in the embodiment of the present invention. Vector and column vector lengths are equal to

In the embodiment of the present invention, the first key value element and the second key value element have various forms, and are optional, and may be in the following forms:

The first sub-key value element includes a key value element consisting of all odd-numbered bits when any of the key-value elements are represented in binary, and the second sub-key value element includes a key-valued element consisting of all even-numbered bits when the binary-valued element is represented in binary.

The key value elements of all odd digits may be decimal, and the key value elements of all even digits may be decimal. Of course, other hexadecimal digits may also be used, and details are not described herein.

For example, when the key element is 37348, 37348 is expressed in binary: 1001000111100100, all odd bits are: 01011010, all even digits are: 10001100, all odd digits represent 90 decimal numbers (first subkey element), all even numbers The decimal number represented by the bit is 140 (the second sub-key element).

Or it can be in the form of:

The first sub-key value element includes a key value element consisting of a 1st to a Kth bit when any of the key value elements are expressed in binary, and the second sub-key value element includes a K+1th bit to the A key element composed of N bits, N is the number of bits when any of the key elements are expressed in binary, 1 ≤ K ≤ N, and K is a positive integer.

For example, when the key element is 37348, 37348 is represented by binary: 1001000111100100, the 0th to 7th digits are: 10010001, the 8th to 15th digits are: 11100100, and the 0th to 7th digits are represented by a decimal number of 90 ( The first sub-key element), the eighth to fifteenth digits represent a decimal number of 140 (the second sub-key element).

In the embodiment of the present invention, after the two-dimensional matrix including the row vector and the column vector is established, the method further includes: initializing an element determined by any one of the two-dimensional matrix and an arbitrary one of the columns into the first preset identifier.

Embodiment 2

Referring to FIG. 3, in the embodiment of the present invention, the detailed process of querying the key element by using the two-dimensional filter generated in FIG. 2 is as follows:

Step 300: Determine, from the hash function set, a hash function subset corresponding to each group of key value elements respectively;

Step 310: Calculate a hash value according to a hash function sub-set corresponding to the key element group of the key element element for each key value element of each group of key value elements, and calculate the hash value in a two-dimensional matrix. The element corresponding to the position in the middle is set as the second preset identifier;

Step 320: Determine, for the key value element to be queried, a hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, and calculate a key value element to be queried according to the corresponding hash function sub-set Greek value

Step 330: Acquire an element corresponding to a position of a hash value of the key element to be queried in the two-dimensional matrix;

Step 340: When the acquired element is the second preset identifier corresponding to the position of the hash value of the key value element to be queried in the two-dimensional matrix, it is determined that the key value element to be queried belongs to the two-dimensional filter representation. A collection of key-valued elements.

If the hash function sub-sets of the two different types of key-value elements are the same, the position of the two-dimensional filter is the same when the two-dimensional filter is loaded, and the different key-value elements correspond to the same position of the two-dimensional filter, so that the query is accurate. In the embodiment of the present invention, the hash function included in the hash function sub-set corresponding to any two different key element groups is different in the embodiment of the present invention; or

For example, the first key value element group is a sales table for the region, and the second key value element group is a sales table for the month, then the hash function sub-set and the second key value element corresponding to the first key value element group The subset of hash functions corresponding to the group is not the same.

In the embodiment of the present invention, there are multiple ways to calculate the hash value of the key element to be queried. Alternatively, the following manner may be adopted:

The first hash value of the key element of the key value element to be queried is calculated according to the hash function sub-set corresponding to the key element group to which the key value element to be queried belongs;

The second hash value is calculated based on the hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, and the second sub-key value element of the key element to be queried.

Of course, the first sub-key element here may also include a key element composed of all the odd bits of the binary element when represented by a binary, and the second sub-key element includes all the even bits of the binary element when represented by a binary representation. Key element; or

Here the specific representation of the first sub-key value element is the same as the representation of the first sub-key value element in the embodiment.

In the embodiment of the present invention, there are multiple ways to obtain the elements of the key value of the key value element to be queried in the two-dimensional matrix. Alternatively, the following manner may be adopted:

Obtaining an element in the two-dimensional matrix with the first hash value as the row and the second hash value as the column; or, obtaining the element in the two-dimensional matrix with the second hash value as the row and the first hash value as the column.

For example, the first sub-key element of any key-value element is 90, the second sub-key element is 140, and the hash function sub-set corresponding to any of the key-value elements is (h1, h2, h3), (h1, h2, h3) The first hash value calculated for 90 is 6, 128, 55, respectively (h1, h2, h3) The second hash value calculated for 140 is 0, 101, 46, respectively, then any key value element is The positions in the two-dimensional matrix are (6, 0), (6, 101), (6, 46), (128, 0), (128, 101), (128, 46), (55, 0) (55, 101), (55, 46), then the elements corresponding to these positions are set to the second preset identifier, or the position of any key element in the two-dimensional matrix is (0, 6), (101, 6), (46, 4), (0, 128), (101, 128), (46, 128), (0, 55), (101, 55), (46, 55), then these positions are corresponding The elements are all set to the second preset identifier.

In the second embodiment, when the query key value element belongs to the key value element included in the plurality of key value element groups, only the two-dimensional filter query is needed, and it is not necessary to generate a corresponding one for each key value element group. The Bloom filter, and when querying whether the key element belongs to a key element included in a plurality of key element groups, it is not necessary to separately query based on multiple Bloom filters, thereby solving the current query efficiency. Lower defects.

The two-dimensional matrix is described as an example in the first embodiment and the second embodiment. Of course, it can also be a multi-dimensional matrix such as a three-dimensional matrix or a four-dimensional matrix. The process of generating a multi-dimensional matrix is similar to the process of generating a two-dimensional matrix, and is based on The query process of the multi-dimensional matrix is similar to the query process based on the two-dimensional matrix, and will not be described in detail here.

In order to better understand the embodiments of the present invention, a specific application scenario is given below, and a detailed description is made for the process of querying key element elements, as shown in FIG. 4:

Embodiment 3

Step 400: Establish a two-dimensional matrix including three row vectors and three column vectors;

Step 410: Determine a hash function set, and generate a two-dimensional filter including a two-dimensional matrix and a hash function set;

Wherein each hash function in the hash function set corresponds to at least one key value element group, and any one of the hash function sets in the hash function set corresponds to any one of the at least one key value element group A subkey element is hashed to obtain a first hash value, and a second hash value element of any key element element in the corresponding key value element group is hashed to obtain a second hash value, the first hash value a positive integer that is less than or equal to the length of the row vector, and the second hash value is a positive integer that is less than or equal to the length of the column vector;

And the set of hash functions determined in this step includes 10 hash functions: h1, h2, h3, h4, h5, h6, h7, h8, h9, h10.

Step 420: Initialize an element determined by any one row vector and any one column vector in the two-dimensional matrix into a first preset identifier;

Step 430: Determine, from the determined set of hash functions, a subset of the hash function corresponding to the two sets of key element elements respectively;

Step 440: Calculate a hash value according to a hash function sub-set corresponding to the key element group of the key element element for any one of the two key value element groups, and calculate the hash value. The element corresponding to the position in the two-dimensional matrix is preset as the second preset identifier;

Step 450: Determine, for the key value element to be queried, a hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, and calculate a key value element to be queried according to the corresponding hash function sub-set Greek value

Step 460: Acquire an element corresponding to a position of a hash value of the key element to be queried in the two-dimensional matrix;

Step 470: Determine whether the acquired element is a second preset identifier corresponding to a position of the hash value of the key element to be queried in the two-dimensional matrix, and if yes, determine that the key value element to be queried belongs to the two-dimensional filter. The set of key-valued elements represented by the device; otherwise, it is determined that the key-valued element to be queried does not belong to the set of key-valued elements represented by the two-dimensional filter.

Based on the foregoing technical solution, referring to FIG. 5, an embodiment of the present invention provides a device for generating a two-dimensional filter, where the generating device includes an establishing unit 50, a determining unit 51, and a generating unit 52, where:

An establishing unit 50, configured to establish a two-dimensional matrix including at least two row vectors and at least two column vectors;

a determining unit 51, configured to determine a hash function set, wherein each hash function in the hash function set corresponds to at least one key value element group, and any one of the hash function pairs in the hash function set corresponds to at least The first sub-key value element of any key-value element in a key-value element group is hashed to obtain a first hash value, and the second sub-key value element of any key-value element in the corresponding key-value element group is hashed Obtaining a second hash value, each of which is a positive integer less than or equal to the length of the row vector, and the second hash value is a positive integer less than or equal to the length of the column vector;

The generating unit 52 is configured to generate a two-dimensional filter including a two-dimensional matrix and a hash function set.

In the embodiment of the present invention, optionally, the length of the row vector and the length of the column vector included in the two-dimensional matrix generated by the establishing unit 50 are greater than or equal to

Where Sr is the number of all key element elements included in all key value element groups; or, Sr is the number of key value elements obtained by filtering all query key conditions of all key value elements included in all key value element groups.

In the embodiment of the present invention, optionally, the first sub-key value element obtained by the determining unit 51 includes a key value element composed of all the odd-numbered bits when any of the key-value elements are expressed in binary, and the second sub-key value element obtained by the determining unit 51 is determined. a key-value element consisting of all even-numbered bits when any key-valued element is represented in binary; or

The first sub-key value element obtained by the determining unit 51 includes a key value element composed of the 1st to the Kth bits when any of the key value elements are expressed in binary, and the second sub-key value element obtained by the determining unit 51 includes any key value element in binary. The key element composed of the K+1th to the Nthth position at the time of representation, and N is the number of bits when any of the key elements is expressed in binary, 1≤K≤N, and K is a positive integer.

In the embodiment of the present invention, the method further includes an initialization unit 53 for initializing an element determined by any one of the two-dimensional matrix and any one of the column vectors as the first preset identifier.

As shown in FIG. 6 , it is a physical device diagram of a two-dimensional filter generating apparatus provided by the present invention. The two-dimensional filter generating apparatus includes at least one processor 601, a communication bus 602, a memory 603, and at least one communication interface 604.

The communication bus 602 is used to implement the connection and communication between the above components, and the communication interface 604 is used to connect and communicate with external devices.

The memory 603 is used to store program code that needs to be executed. When the processor 601 executes the program code in the memory 603, the following functions are implemented:

Determining a hash function set, wherein each hash function in the hash function set corresponds to at least one key value element group, and any one of the hash function pairs in the hash function set corresponds to at least one key value element group The first sub-key value element of any of the key element elements is hashed to obtain a first hash value, and the second sub-key value element of any key element element in the corresponding key value element group is hashed to obtain a second hash value. The first hash value is a positive integer less than or equal to the length of the row vector, and the second hash value is a positive integer less than or equal to the length of the column vector;

A two-dimensional filter is generated that includes a two-dimensional matrix and a collection of hash functions.

Based on the foregoing technical solution, as shown in FIG. 7, an embodiment of the present invention provides an apparatus for querying a key value element, where the apparatus for querying a key value element includes a determining unit 70, a setting unit 71, a calculating unit 72, an obtaining unit 73, and a query. Unit 74, wherein:

a determining unit 70, configured to determine, from the hash function set, a hash function subset corresponding to each group of key value elements;

The setting unit 71 is configured to calculate, according to any one of the key value elements of each set of key value elements, a hash value according to the hash function subset corresponding to the key element element group to which the key value element belongs, and calculate the hash value in The element corresponding to the position in the two-dimensional matrix is set as the second preset identifier;

The calculating unit 72 is configured to determine, according to the key value element to be queried, a hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, and calculate a key value to be queried according to the corresponding hash function sub-set The hash value of the element;

The obtaining unit 73 is configured to obtain an element corresponding to a position of the hash value of the key element group to be queried in the two-dimensional matrix;

The query unit 74 is configured to: when the acquired element is a second preset identifier corresponding to a position of the hash value of the key value element to be queried in the two-dimensional matrix, determine that the key value element to be queried belongs to the two-dimensional filtering A collection of key-valued elements represented by the device.

In the embodiment of the present invention, optionally, the hash function included in the hash function sub-set corresponding to any two different key element group determined by the determining unit 70 is different; or

The hash function included in the hash function sub-set corresponding to any two different key element group determined by the determining unit 70 is the same, and the hash function is arranged differently.

In the embodiment of the present invention, optionally, the calculating unit 72 is specifically configured to:

In the embodiment of the present invention, the obtaining unit 73 is specifically configured to:

As shown in FIG. 8 , it is a physical device diagram of a two-dimensional filter generating apparatus provided by the present invention. The two-dimensional filter generating apparatus includes at least one processor 801, a communication bus 802, a memory 803, and at least one communication interface 804.

The communication bus 802 is used to implement the connection and communication between the above components, and the communication interface 804 is used to connect and communicate with external devices.

The memory 803 is configured to store program code that needs to be executed. When the processor 801 executes the program code in the memory 803, the following functions are implemented:

Determining, from the set of hash functions, a subset of hash functions corresponding to each set of key value elements;

For each key value element of each set of key value elements, a hash value is calculated according to the hash function sub-set corresponding to the key value element group of the key value element, and the calculated hash value is in the two-dimensional matrix. The element corresponding to the location is set as the second preset identifier;

Determining, according to the key value element to be queried, a hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, and calculating a hash value of the key value element to be queried according to the corresponding hash function sub-set;

Obtaining an element corresponding to the position of the hash value of the key element to be queried in the two-dimensional matrix;

When the acquired element is the second preset identifier corresponding to the position of the hash value of the key element to be queried in the two-dimensional matrix, it is determined that the key value element to be queried belongs to the key value represented by the two-dimensional filter Collection of elements.

In summary, in the embodiment of the present invention, a two-dimensional filter includes a two-dimensional matrix, and the two-dimensional matrix can be linked with a plurality of key element groups, thereby improving the flexibility of the filter.

Further, when the query key value element belongs to the key value element included in the plurality of key value element groups, only the two-dimensional filter query is needed, and it is not necessary to generate the Bronze corresponding to each of the key value element groups respectively. The filter, and when the query key value element belongs to the key value element included in the plurality of key value element groups, it is not necessary to separately query based on the multiple bloom filters, thereby solving the current query efficiency is low. defect.

The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (system), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions in one or more of the flow or in a block or blocks of the flowchart.

The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The apparatus functions in one or more blocks of a flow or a flow diagram and/or block diagram of a flowchart.

These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions in one or more blocks of the flowchart or in a flow or block of the flowchart.

While the preferred embodiment of the invention has been described, it will be understood that Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and the modifications and

It is apparent that those skilled in the art can make various modifications and variations to the embodiments of the invention without departing from the spirit and scope of the embodiments of the invention. Thus, it is intended that the present invention cover the modifications and modifications of the embodiments of the invention.

Claims

A method for querying a key value element, comprising:

Determining, from the hash function set, a hash function sub-set corresponding to each group of key value element groups;

For each key value element in each group of key value element groups, a hash value is calculated according to the hash function sub-set corresponding to the key value element group to which the key value element belongs, and the calculated hash value is in the two-dimensional matrix. The element corresponding to the location is set as the second preset identifier;

Determining, according to the key value element to be queried, a hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, and calculating the key value to be queried according to the corresponding hash function sub-set The hash value of the element;

Obtaining an element corresponding to a position of the hash value of the to-be-queried key value element in the two-dimensional matrix;

Determining that the key value element to be queried belongs to two-dimensional when the acquired element is a second preset identifier corresponding to a position of the hash value of the to-be-queried key element in the two-dimensional matrix. A collection of key-valued elements represented by the filter.
The method according to claim 1, wherein the hash function included in the hash function sub-set corresponding to any two different key element groups is different; or

The hash function included in the hash function sub-set corresponding to any two different key element groups is the same, and the hash function is arranged differently.
The method of claim 1 or 2, wherein calculating the hash value of the key element to be queried comprises:

Calculating, according to the hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, calculating a first hash value for the first sub-key value element of the key value element to be queried;

And calculating, according to the hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, a second hash value is calculated for the second sub-key value element of the key value element to be queried.
The method of claim 3, wherein the key element to be queried is obtained The element corresponding to the location of the hash value in the two-dimensional matrix includes:

Obtaining, in the two-dimensional matrix, an element having the first hash value as a row and the second hash value as a column; or acquiring the second hash value in the two-dimensional matrix, The first hash value is an element of the column.
A method for generating a two-dimensional filter, comprising:

Establishing a two-dimensional matrix comprising at least two row vectors and at least two column vectors;

Determining a set of hash functions, wherein each hash function in the set of hash functions corresponds to at least one group of key value elements, and any one of the hash function sets corresponds to at least one key The first sub-key value element of any key value element in the value element group is hashed to obtain a first hash value, and the second sub-key value element of the any key value element is hashed to obtain a second hash value, The first hash value is a positive integer that is less than or equal to the length of the row vector, and the second hash value is a positive integer that is less than or equal to the length of the column vector;

A two-dimensional filter including the two-dimensional matrix and the set of hash functions is generated.
The method of claim 5 wherein the length of the row vector and the length of the column vector are greater than or equal to

Wherein, Sr is the number of all the key element elements included in all the key element group; or Sr is the number of key element elements obtained by filtering all the key value elements included in the all key element group.
A method according to claim 5 or claim 6, wherein said first sub-key value element comprises a key value element consisting of all odd-numbered bits when said any of said key-value elements are represented in binary, said second sub-key value The element includes a key element consisting of all the even bits of any of the key element elements represented in binary; or

The first sub-key value element includes a key value element consisting of a first bit to a Kth bit when the any of the key value elements is expressed in binary, and the second sub-key value element includes when the any of the key value elements is expressed in binary The key value element consisting of the K+1th to the Nth bits, N is the number of bits when any of the key elements is expressed in binary, 1≤K≤N, and K is a positive integer.
The method of any of claims 5-7, further comprising:

An element determined by any one of the two-dimensional matrix and any one of the columns of vectors is initialized to a first preset identifier.
A device for querying a key value element, comprising:

a determining unit, configured to determine, from the hash function set, a hash function sub-collection corresponding to each group of key value element groups;

a setting unit, configured to calculate a hash value according to a hash function sub-set corresponding to the key value element group of the key value element group for each key value element group in each group of key value element groups, and calculate the hash value in The element corresponding to the position in the two-dimensional matrix is set as the second preset identifier;

a calculation unit, configured to determine, according to the key value element to be queried, a hash function subset corresponding to the key element group to which the key value element to be queried belongs, and calculate the hash function according to the corresponding hash function subset The hash value of the key element to be queried;

An acquiring unit, configured to acquire an element corresponding to a location of the hash value of the group of key element elements to be queried in the two-dimensional matrix;

a query unit, configured to determine, when the acquired element is the second preset identifier corresponding to a location of the hash value of the to-be-queried key element in the two-dimensional matrix, The key-value element belongs to a collection of key-valued elements represented by a two-dimensional filter.
The apparatus according to claim 9, wherein the hash function included in the hash function sub-set corresponding to any two different key element group determined by the determining unit is different; or

The hash function included in the hash function sub-set corresponding to any two different key element group determined by the determining unit is the same, and the hash function is arranged differently.
The device according to claim 9 or 10, wherein the calculating unit is specifically configured to:

Calculating, according to the hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, calculating a first hash value for the first sub-key value element of the key value element to be queried;

And calculating, according to the hash function sub-set corresponding to the key element group to which the key value element to be queried belongs, a second hash value is calculated for the second sub-key value element of the key value element to be queried.
The device according to claim 11, wherein the obtaining unit is specifically configured to:

Obtaining, in the two-dimensional matrix, an element having the first hash value as a row and the second hash value as a column; or acquiring the second hash value in the two-dimensional matrix, The first hash value is an element of the column.
A device for generating a two-dimensional filter, comprising:

Establishing a unit for establishing a two-dimensional matrix comprising at least two row vectors and at least two column vectors;

a determining unit, configured to determine a hash function set, wherein each hash function in the hash function set corresponds to at least one key value element group, and any one of the hash function sets The first sub-key value element of any one of the corresponding at least one key value element group is hashed to obtain a first hash value, and the second sub-key value element of the any of the key value elements is hashed to obtain a a second hash value, the first hash value being a positive integer less than or equal to the length of the row vector, and the second hash value being a positive integer less than or equal to the length of the column vector;

And a generating unit, configured to generate a two-dimensional filter including the two-dimensional matrix and the hash function set.
The apparatus according to claim 13, wherein the length of the row vector included in the two-dimensional matrix generated by the establishing unit and the length of the column vector are greater than or equal to

Wherein, Sr is the number of all the key element elements included in all the key element group; or Sr is the number of key element elements obtained by filtering all the key value elements included in the all key element group.
The apparatus according to claim 13 or 14, wherein said first sub-key value element obtained by said determining unit comprises a key value element consisting of all odd-numbered bits when said one of said key-value elements is represented in binary, said determining The second sub-key value element obtained by the unit includes a key value element consisting of all the even bits of the binary key element when the binary value element is represented; or

The first sub-key value element obtained by the determining unit includes a key value element composed of the first to the Kth bits when the any key value element is expressed in binary, and the second sub-key value element obtained by the determining unit includes the A key value element consisting of a K+1th bit to an Nth bit when any of the key value elements is expressed in binary, and N is a bit number when the any of the key value elements is expressed in binary, 1≤K≤N, and K is a positive integer.
The apparatus according to any one of claims 13-15, further comprising an initialization unit, configured to initialize an element determined by any one of the two-dimensional matrix and an arbitrary one of the columns to the first preset Logo.