KR20210089589A - Method for processing bloom filter using circular shift, and apparatuses for performing the same - Google Patents

Abstract

Disclosed are a method for processing a bloom filter using a cyclic shift and devices for performing the same. The method for processing the bloom filter according to one embodiment may comprise: an operation of generating a hash value for an element using one hash function; an operation of selecting and outputting a first specific bit from the hash value by performing a cyclic shift on the hash value; and an operation of performing an operation of adding or querying for the element to a bit array of a bloom filter based on the first specific bit. Therefore, the present invention is capable of providing a technology for improving the operation speed.

Description

A method of processing a bloom filter using a cyclic shift and apparatuses for performing the same {METHOD FOR PROCESSING BLOOM FILTER USING CIRCULAR SHIFT, AND APPARATUSES FOR PERFORMING THE SAME}

The disclosure below relates to a method of processing a bloom filter using a cyclic shift and apparatuses for performing the same.

The bloom filter, devised by B. Bloom in 1970, is a probabilistic data structure that can check whether an element belongs to a set. In case of externally checking whether some data is included in the set, the index of the hash values by k different hash functions of the data to be checked is obtained. If all bits in the corresponding positions are 1, it is positively determined that the data to be tested belongs to the set, and if any of those bits is 0, it is negatively determined that it does not belong to the set.

Depending on the inevitable imperfection of a hash function that maps a set to another set of a smaller size, hash values of some data that do not belong to a set can be derived identically to the hash values of elements belonging to the set. Since the bloom filter entirely depends on the hash value, if the hash value is the same, data not belonging to the set and the elements belonging to the set cannot be distinguished by the hash function, and data not belonging to the set can be determined to belong to the set. have.

Although the bloom filter determines that an element belongs to a set positively, the possibility of false positives that the element does not actually belong to the set is a problem. There is absolutely no possibility of false negatives that might belong to the set.

Depending on the design, the bloom filter consumes a small memory space, has a fast determination speed, and is not capable of false negatives. It can accurately and quickly screen small databases.

Bloom filters are widely used in various database applications and large-scale storage applications, and are increasingly frequently applied in various network fields such as routing table search, online traffic control, and intrusion detection systems.

The background art described above is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and it cannot necessarily be said to be a known technology disclosed to the general public before the present application.

Embodiments may provide a technique for improving computation speed by using fewer computations than conventional Bloom filters with similar false positives to conventional Bloom filters.

However, the technical problems are not limited to the above-described technical problems, and other technical problems may exist.

A method of processing a bloom filter according to an embodiment includes generating a hash value for an element using a single hash function; performing a cyclic shift on the hash value to select and output a first specific bit from the hash value; and performing an operation of adding or querying the element to a bit array of a bloom filter based on the first specific bit.

The outputting operation may include outputting the first specific bit as an index of a bit array of the bloom filter.

The cyclic shift may be a cyclic bit shift.

The outputting operation may include an operation of selecting and outputting the first specific bit after right shifting by a second specific bit in the hash value whenever a cyclic bit shift is performed.

The first specific bit may be a continuous bit.

The size of the first specific bit may be determined based on the size of the bloom filter.

The cyclic shift may be implemented in assembly instructions.

An apparatus for processing a bloom filter according to an embodiment includes a memory for storing one or more instructions; and a processor for executing the instruction, wherein when the instruction is executed, the processor generates a hash value for an element using a single hash function, and performs a cyclic shift on the hash value to obtain the hash value An operation of adding or querying the element to the bit array of the bloom filter may be performed by selecting and outputting a first specific bit from the first specific bit.

The processor may output the first specific bit as an index of the bit array of the bloom filter.

The cyclic shift may be a cyclic bit shift.

The processor may select and output the first specific bit after right shifting by a second specific bit in the hash value whenever a cyclic bit shift is performed.

The first specific bit may be a continuous bit.

The size of the first specific bit may be determined based on the size of the bloom filter.

The cyclic shift may be implemented in assembly instructions.

1 is a block diagram of an electronic device according to an exemplary embodiment.
2 is a view for explaining the characteristics of the bloom filter.
3 is an example for explaining an operation of simulating a hash function using a cyclic shift according to an embodiment.
4 shows another example for explaining an operation of simulating a hash function using a cyclic shift according to an embodiment.
5 shows an example of pseudocode expressing a cyclic shift in an assembly instruction.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the actual implemented form is not limited to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, these terms should be interpreted only for the purpose of distinguishing one component from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

1 is a block diagram of an electronic device according to an exemplary embodiment.

The electronic device 100 may process data using a Bloom filter (eg, the Bloom filter 200 of FIG. 2 ). The electronic device 100 may use the bloom filter 200 to add the existence of data or to check the existence of data. Bloom filter 200 may be a hash-based data structure used to check whether an element belongs to a set. The bloom filter 200 may be used as an index to check the existence of data due to an increase in data stored in a cache memory, a routing table, etc. stored in the system. In addition, the bloom filter 200 may be used for various purposes, such as a Squid web proxy cache, resource routing, packet routing, or malicious URL identification.

In processing the Bloom filter, the electronic device 100 performs a cyclic shift operation (eg, a cyclic bit shift operation) while using a single hash function, thereby having the same level of false positive probability as the existing Bloom filter and faster operation speed. operation can be performed with

The electronic device 100 may be implemented as a personal computer (PC), a data server, or a portable device. Portable electronic devices include a laptop computer, a mobile phone, a smart phone, a tablet PC, a mobile internet device (MID), a personal digital assistant (PDA), and an enterprise digital assistant (EDA). ), digital still camera, digital video camera, PMP (portable multimedia player), PND (personal navigation device or portable navigation device), handheld game console, e-book (e-book) may be implemented as a smart device. In this case, the smart device may be implemented as a smart watch or a smart band.

The electronic device 100 may include a memory 110 and a processor 130 .

The memory 110 may store instructions (or programs) executable by the processor 130 . For example, the instructions may include instructions for executing an operation of the processor 130 and/or an operation of each component of the processor 130 .

The processor 130 may process data stored in the memory 110 . The processor 130 may execute computer readable codes (eg, software) stored in the memory 110 and instructions induced by the processor 130 .

The processor 130 may be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program.

For example, a data processing device implemented as hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

In processing the Bloom filter, the processor 130 may perform a cyclic shift operation (eg, a cyclic bit shift operation) while using one hash function for the Bloom filter. Hereinafter, this will be described in detail.

2 is a view for explaining the characteristics of the bloom filter. In FIG. 2 , n means the size of an input element, m means the size of a bloom filter, k means the number of hash functions, and h( ) means a hash function.

The bloom filter 200 may be a very useful structure for space utilization of data by setting k hash function result values for an input element to '1' in the corresponding bit of an array having a size of m.

는 입력 집합 S =

의 각 원소를 해시하여 해시 값들을 생성하며, 해시 값들은 블룸 필터(200)의 인덱스(예: 주소, 주소 번호)를 나타낸다.The operation for the bloom filter 200 may include an add operation (eg, an element addition operation) and a query operation (eg, an element check operation (membership test operation)). A hash function is required for the operation on the bloom filter 200 . k independent random hash functions

is the input set S =

Hash values are generated by hashing each element of , and the hash values indicate an index (eg, address, address number) of the bloom filter 200 .

_,

를 블룸 필터(200)에 대해 추가하는 동작을 도시하고 있다. 초기에, m(예: 10) 비트의 블룸 필터(200)는 모두 '0'으로 설정될 수 있다. k개의 해시 함수들이 입력 원소(예: 추가하려는 원소)에 대해서 k개의 해시 값을 계산하여 출력할 수 있다. 이때, 각 해시 값은 블룸 필터(200)의 비트 배열의 인덱스일 수 있다. 블룸 필터(200)에서 각 해시 값에 대응하는 비트는 1로 설정될 수 있다.2, the input element

_,

The operation of adding to the bloom filter 200 is shown. Initially, all m (eg, 10) bit bloom filters 200 may be set to '0'. k hash functions can calculate and output k hash values for an input element (eg, an element to be added). In this case, each hash value may be an index of the bit array of the bloom filter 200 . A bit corresponding to each hash value in the bloom filter 200 may be set to 1.

에 대한 세 개의 해시 함수의 해시 값 각각이 제1 비트(201)(예: 블룸 필터(200)의 비트 배열의 1번째 비트), 제1 비트(201), 및 제5 비트(205)(예: 블룸 필터(200)의 비트 배열의 5번째 비트)를 가리키는 것이라면, 이에 해당되는 제1 비트(201), 제5 비트(205)가 '1'로 설정될 수 있다. 마찬가지로, 원소

에 대해서는 제6 비트(206)(예: 블룸 필터(200)의 비트 배열의 6번째 비트), 제7 비트(207)(예: 블룸 필터(200)의 비트 배열의 7번째 비트), 제9 비트(209)(예: 블룸 필터(200)의 비트 배열의 9번째 비트)가 '1'로 설정될 수 있다. 이러한 블룸 필터(200)에는 원소

과

가 존재함을 표시하고 있다.For example, element

Each of the hash values of the three hash functions for , is the first bit 201 (eg, the first bit of the bit array of the bloom filter 200), the first bit 201, and the fifth bit 205 (eg : 5th bit of the bit array of the bloom filter 200), the corresponding first bit 201 and the fifth bit 205 may be set to '1'. Likewise, the element

6th bit 206 (eg the 6th bit of the bit array of the bloom filter 200), the 7th bit 207 (eg the 7th bit of the bit array of the bloom filter 200), the ninth bit 206 for The bit 209 (eg, the ninth bit of the bit array of the bloom filter 200) may be set to '1'. The bloom filter 200 has an element

and

indicates the existence of

,

를 블룸 필터(200)에 대해 쿼링하는 동작을 도시하고 있다. 쿼리 동작은 블룸 필터(200)에 대해 원소의 존재 여부를 확인하기 위한 동작일 수 있다. k개의 해시 함수들이 입력 원소(예: 검사하려는 원소)에 대해서 k개의 해시 값을 계산하여 출력할 수 있다. 블룸 필터(200)에서 각 해시값에 대응하는 비트에 설정된 값을 읽고, k개의 비트가 모두 1인 경우 원소는 집합에 속한다고 판단하고, 그렇지 않은 경우 속하지 않는다고 판단할 수 있다.2, the input element

,

The operation of querying for the bloom filter 200 is shown. The query operation may be an operation for checking whether an element exists with respect to the bloom filter 200 . k hash functions can compute and output k hash values for an input element (eg, an element to be checked). The bloom filter 200 reads the value set in the bit corresponding to each hash value, and when all k bits are 1, it is determined that the element belongs to the set, otherwise it can be determined that it does not belong.

의 경우는 실제 존재하는 개체이므로 true positive라 하고, 반면에 추가되지 않은 원소

에 대해서는 실제로 블룸 필터(200)에 존재하지 않지만, 블룸 필터(200)의 쿼리 동작을 통하여 존재하는 것으로 표현되기 때문에 이를 긍정오류(false positive)라 한다.For example, elements normally added for bloom filter 200

In the case of , it is said to be true positive because it is an entity that actually exists, while elements that are not added

Although it does not actually exist in the bloom filter 200, since it is expressed as being present through the query operation of the bloom filter 200, this is called a false positive.

In this way, the bloom filter 200 can quickly and efficiently search for the existence of a large amount of data. That is, since long information is shortened and stored as a bit flag, the existence of data is determined using the corresponding information.

A positive error occurring due to the characteristics of the bloom filter 200 will be described through the following mathematical procedures.

Equation 1 may be a probability of a specific bit that is not set to 1 when an element is stored in the bloom filter 200 of size m.

[Equation 1]

Since k hash functions generate k outputs (eg, hash values), Equation 2 may be a probability of not setting '1' to a bit k times.

[Equation 1]

When n inputs are stored in the Bloom filter 200, the probability of being set to 0 when n inputs are stored in the Bloom filter 200 by k hash functions may be equal to Equation 3.

[Equation 3]

(입력ㆍ해시 함수의 수) 출력이 블룸 필터(200)에 저장될 때, 블룸 필터(200)에 0이 아닌 비트(예: 1)가 설정될 확률일 수 있다.Equation 4 is

(Number of input/hash functions) When the output is stored in the bloom filter 200 , it may be a probability that a non-zero bit (eg, 1) is set in the bloom filter 200 .

[Equation 4]

After k hash functions hash the element, k hash values may be inserted into the bloom filter 200 . Accordingly, Equation 5 represents a theoretical false positive rate.

[Equation 5]

Also, Equation 5 may be calculated as a rough false positive probability as in Equation 6 .

[Equation 6]

Therefore, the minimum value of a false positive (eg, false positive probability) may be the same as in Equation 7.

[Equation 7]

It can be seen that the false positive probability can be increased by 1) more hash functions 2) an increase in input elements 3) a decrease in the size of the bloom filter.

The number of hash functions in the bloom filter 200 is related to the calculation speed. The embodiments use one hash function to maintain the false positive probability of the bloom filter 200 at a level similar to that of an existing bloom filter (eg, a standard bloom filter, a double hash bloom filter, a single hash bloom filter, etc.) while maintaining the bloom filter ( 200) can improve the operation speed.

3 is an example for explaining an operation of simulating a hash function using a cyclic shift according to an embodiment.

In FIG. 3 , for convenience of explanation, it is assumed that X hash functions are used to perform an operation on the bloom filter 200 .

를 이용하여 수행될 수 있다. According to various embodiments, the operation (eg, element addition operation, element check operation) for the bloom filter 200 is not X hash functions, but one hash function (eg, non-cryptographic hash function, cryptographic hash function)

can be performed using

는 입력

으로부터 이진 비트로 표현되는 해시 값

을 생성할 수 있다. 해시 값

에는 순환 비트 시프트(예: X번의 순환 비트 시프트)가 수행될 수 있다. 순환 비트 시프트가 수행될 때마다, 해시 값

에서 i비트(i = 1,??, X)만큼 오른쪽으로 시프트된 후 연속되는 특정 비트(예: 특정 비트 배열)가 선택되어 출력될 수 있다.hash function

is input

Hash value expressed in binary bits from

can create hash value

A cyclic bit shift (eg, cyclic bit shift X times) may be performed. Whenever a cyclic bit shift is performed, the hash value

After shifting to the right by i bits (i = 1,??, X), a continuous specific bit (eg, a specific bit array) may be selected and output.

에서 제1 특정 비트(301)가 선택되어 제1 해시 함수의 해시 값

으로 출력되고, 제2 순환 비트 시프트가 수행된 경우 해시 값

에서 제2 특정 비트(302)가 선택되어 제2 해시 함수의 해시 값

으로 출력되고, 제3 순환 비트 시프트가 수행된 경우 해시 값

에서 제3 특정 비트(303)가 선택되어 제3 해시 함수의 해시 값

으로 출력되고, 제4 순환 비트 시프트가 수행된 경우 해시 값

에서 제4 특정 비트(304)가 선택되어 제4 해시 함수의 해시 값

으로 출력될 수 있다. 총 X번(예: 시뮬레이션하려는 해시 함수의 개수) 순환 시프트 방법을 통하여서 하나의 해시 함수의 해시 값에서 각각 다른 해시 함수의 해시 값을 만들어낼 수 있다.For example, if the first cyclic bit shift is performed, the hash value

The first specific bit 301 is selected in the hash value of the first hash function

, and a hash value when the second cyclic bit shift is performed

The second specific bit 302 is selected in the hash value of the second hash function

, and a hash value when the third cyclic bit shift is performed

The third specific bit 303 is selected in the hash value of the third hash function

, and a hash value when the fourth cyclic bit shift is performed

A fourth specific bit 304 is selected in the hash value of the fourth hash function

can be output as A total of X times (eg, the number of hash functions to be simulated) can generate hash values of different hash functions from the hash values of one hash function through the cyclic shift method.

The size of the i-bit shifted by the cyclic bit shift and the size of the selected and output specific bit may be determined based on the size of the bloom filter 200 .

4 shows another example for explaining an operation of simulating a hash function using a cyclic shift according to an embodiment, and FIG. 5 shows an example of pseudocode expressing the cyclic shift by an assembly instruction.

Equation 8 may represent the description of the cyclic bit shift as a high-level programming language.

[Equation 8]

Equation 8 may represent a pseudo code in which bits are shifted to the right by the i-th (i = 1,..., X) address in every loop. 4 may more intuitively show Equation (8). When the expression is applied to the implementation of the bloom filter 200, a problem may occur in Equation 8, and a selection operation for selecting a specific bit is required. Assembly language is needed to reduce the corresponding computation time. To implement a cyclic bit shift using the CF flag, ROL or ROR (short for Rotate Right and Rotate Left) assembly instructions can be used to reduce computation time.

5 shows an example of a pseudo code of a cyclic bit shift performed in the bloom filter 200 . The highlighted line contains ROL/ROR. As can be seen from Equation 8, cyclic bit shifting of the hash value by i bits can reduce several tasks with one assembly instruction. The initial hash output d can be cyclically rotated per loop to simulate an arbitrary hash value. To be compatible with c/c++, there can be '__asm' blocks of code in the middle of high-level programming languages.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and a software application running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc. alone or in combination, and the program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. have. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, a person skilled in the art may apply various technical modifications and variations based thereon. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (15)

generating a hash value for an element using a single hash function;
performing a cyclic shift on the hash value to select and output a first specific bit from the hash value; and
performing an operation of adding or querying the element to a bit array of a bloom filter based on the first specific bit
A method of processing a bloom filter, comprising:
According to claim 1,
The output operation is
Outputting the first specific bit as an index of the bit array of the bloom filter
A method of processing a bloom filter, comprising:
According to claim 1,
wherein the cyclic shift is a cyclic bit shift.
According to claim 1,
The output operation is
An operation of selecting and outputting the first specific bit after shifting to the right by a second specific bit in the hash value whenever a cyclic bit shift is performed
A method of processing a bloom filter, comprising:
According to claim 1,
wherein the first specific bit is a continuous bit.
According to claim 1,
and the size of the first specific bit is determined based on the size of the bloom filter.
According to claim 1,
wherein the cyclic shift is implemented in assembly instructions.
A computer program stored in a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 to 7.
An apparatus for processing a bloom filter, comprising:
a memory storing one or more instructions; and
a processor for executing the instructions
including,
When the instruction is executed, the processor,
A hash value is generated for an element using a single hash function, a cyclic shift is performed on the hash value to select and output a first specific bit from the hash value, and the bloom filter is based on the first specific bit. An apparatus for performing an operation to add or query an element to a bit array.
10. The method of claim 9,
The processor is
outputting the first specific bit as an index of the bit array of the bloom filter.
10. The method of claim 9,
wherein the cyclic shift is a cyclic bit shift.
10. The method of claim 9,
The processor is
The apparatus of claim 1, wherein each time a cyclic bit shift is performed, the hash value is shifted to the right by a second specific bit, and then the first specific bit is selected and output.
10. The method of claim 9,
wherein the first specific bit is a contiguous bit.
10. The method of claim 9,
and the size of the first specific bit is determined based on the size of the bloom filter.
10. The method of claim 9,
wherein the cyclic shift is implemented in assembly instructions.

Images