JP5526985B2 - Search program, search device, and search method - Google Patents

Description

  The present invention relates to a search program, a search device, and a search method for searching for data using a Bloom filter.

  Conventionally, when large-scale data is managed in a tree structure, a relatively large amount of data is managed in a data structure called a B-tree. Compared to a simple binary tree, the B-tree stores a plurality of data entries in one block, and therefore has an advantage that the range in which the change in the shape of the tree structure spreads can be narrowed even if data entries are added. is there. For this reason, the B-tree is often used as a data management method for a disk such as a hard disk.

  However, when retrieving data managed in a tree structure on the disk, it is necessary to actually read a plurality of data blocks. In general, I / O (input / output) for a disk is slower than memory access, and thus searching for data on the disk may take time and effort.

  For this reason, recently, in order to avoid a search delay due to disk I / O, a countermeasure such as having a tree structure in the memory has been considered. However, in the B-tree, when the number of data entries increases, there is a possibility that the required memory amount increases accordingly. For this reason, a method using a method (cache) in which only the most read portion of the tree structure is stored in the memory is also considered.

  On the other hand, a data structure called a Bloom filter has recently been known. The Bloom filter is a method for efficiently checking whether an entry belongs to an existing set. In addition, there is also disclosed a group process in which two pulse speed bits and even / odd bits are provided in the dial pulse in the dial pulse process of the electronic exchange, and the bits are fetched.

JP 2007-52698 A Japanese Patent Laid-Open No. 4-18895

  As described above, since the B-tree can handle a large amount of data, it is possible to reduce disk I / O if a cache is appropriately mounted. However, the number of times cannot be reduced beyond a certain level. Further, when the tree structure changes due to the addition of the data entry, I / O for managing the tree structure may be required. In addition, the Bloom filter can only be used for data management because it knows only the existence of a data entry.

  An object of the present invention is to provide a search program, a search device, and a search method capable of speeding up data search when a Bloom filter is used in order to solve the above-described problems caused by the prior art. .

  In order to solve the above-mentioned problems and achieve the object, a data block set including a registered data group for each data block and a Bloom filter in which m bits indicating negative in a predetermined number of data blocks are arranged When the n number of Bloom filter columns arranged in units of the predetermined number of data blocks can be accessed, a transposition request for the Bloom filter column is accepted, and when a transposition request is accepted, the Bloom filter sequence is The data block set is transposed to a transposed Bloom filter sequence in which n-bit transposed Bloom filters in which bit sequences in each Bloom filter are collected by bits at the same position are arranged, and the transposed Bloom filter sequence is used. It is a requirement to determine the presence or absence of data in the network.

  According to the search program, the search device, and the search method according to the present invention, it is possible to increase the speed of data search when using the Bloom filter.

It is a block diagram which shows the hardware structural example of the management apparatus concerning embodiment. It is a block diagram which shows the structural example of the management apparatus concerning this Embodiment. It is explanatory drawing which shows an example of a hash table group. It is explanatory drawing which shows an example of a hierarchical Bloom filter. It is explanatory drawing which shows the learning process example of the hierarchical Bloom filter by a registration process part. It is explanatory drawing which shows the example of a search process of the hierarchical Bloom filter by a search process part. It is explanatory drawing which shows the transposition example of the Bloom filter row | line | column of the p-th stage in a hierarchical Bloom filter. It is explanatory drawing which shows the search processing example of the hierarchical transposed Bloom filter by a search process part. It is a block diagram which shows the functional structural example of a search process part. It is a flowchart which shows the learning processing procedure of the hierarchical Bloom filter by a registration process part. It is a flowchart (the 1) which shows the search processing procedure by a search process part. It is a flowchart (the 2) which shows the search processing procedure by a search process part. It is explanatory drawing which shows the learning process example of the hierarchical transposition Bloom filter tBF by a registration process part. It is a flowchart which shows the learning processing procedure of the hierarchical Bloom filter BF by a registration process part.

  Embodiments of a search program, a search device, and a search method according to the present invention will be described below in detail with reference to the accompanying drawings. First, a configuration example of the management apparatus according to the present embodiment will be described.

<Hardware configuration example of management device>
FIG. 1 is a block diagram of a hardware configuration example of the management apparatus according to the embodiment. In FIG. 1, the search device includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a magnetic disk drive 104, a magnetic disk 105, and an optical disk drive 106. An optical disk 107, a display 108, an I / F (Interface) 109, a keyboard 110, a mouse 111, a scanner 112, and a printer 113. Each component is connected by a bus 100.

  Here, the CPU 101 controls the entire management apparatus. The ROM 102 stores a program such as a boot program. The RAM 103 is used as a work area for the CPU 101. The magnetic disk drive 104 controls reading / writing of data with respect to the magnetic disk 105 according to the control of the CPU 101. The magnetic disk 105 stores data written under the control of the magnetic disk drive 104.

  The optical disk drive 106 controls reading / writing of data with respect to the optical disk 107 according to the control of the CPU 101. The optical disc 107 stores data written under the control of the optical disc drive 106, and causes the computer to read data stored on the optical disc 107.

  The display 108 displays data such as a document, an image, and function information as well as a cursor, an icon, or a tool box. As this display 108, for example, a CRT, a TFT liquid crystal display, a plasma display, or the like can be adopted.

  An interface (hereinafter abbreviated as “I / F”) 109 is connected to a network 114 such as a LAN (Local Area Network), a WAN (Wide Area Network), and the Internet through a communication line. Connected to other devices. The I / F 109 controls an internal interface with the network 114 and controls data input / output from an external device. For example, a modem or a LAN adapter may be employed as the I / F 109.

  The keyboard 110 includes keys for inputting characters, numbers, various instructions, and the like, and inputs data. Moreover, a touch panel type input pad or a numeric keypad may be used. The mouse 111 performs cursor movement, range selection, window movement, size change, and the like. A trackball or a joystick may be used as long as they have the same function as a pointing device.

  The scanner 112 optically reads an image and takes in the image data into the management apparatus. The scanner 112 may have an OCR (Optical Character Reader) function. The printer 113 prints image data and document data. For example, a laser printer or an ink jet printer can be employed as the printer 113.

<Functional configuration example of management device>
FIG. 2 is a block diagram illustrating a configuration example of the management apparatus according to the present embodiment. The management device 200 includes a data block set db, a hash table group HTs, a hierarchical Bloom filter BF, a hierarchical transposed Bloom filter tBF, a registration processing unit 201, and a search processing unit 202.

  The data block set db has a plurality of data blocks, and each data block holds registered data. Each data block is represented as db #. # Is a number indicating the block number. The block number # corresponds to the bit position of the data block db #.

  The hash table group HTs is a set of hash tables corresponding to each data block db # in the data block set db. Each hash table is represented as HT #. # Is a number and matches the block number of the data block db #. The hash table HT # is a table in which a hash value when data is given to a specific hash function is associated with data that is a generation source of the hash value (data itself or a pointer to the data).

  FIG. 3 is an explanatory diagram showing an example of the hash table group HTs. In FIG. 3, SHA-1 is used as the hash function. In addition, what hash function is used may be set in advance.

  Returning to FIG. 2, the hierarchical Bloom filter BF is index information in which the Bloom filter has a hierarchical structure. The hierarchical Bloom filter BF will be described later. The Bloom filter is index information in which a plurality of bits indicating false positives (also called false positives or false positives) / negatives in a predetermined number of data blocks are arranged. When the Bloom filter bit is ON, it indicates false positive, and when it is OFF, it indicates negative. The bit value may be set to 1 and 0 may be set to OFF. Conversely, the bit value may be set to 0 and 1 may be set to OFF. In this embodiment, the bit value is set to 1 and 0 is set to OFF.

  The hierarchical transposed Bloom filter tBF is index information obtained by transposing the hierarchical Bloom filter BF. The hierarchical transposed Bloom filter tBF is generated by the search processing unit 202. Details of the hierarchical transposed blue filter tBF will be described later.

  The data block set db, the hash table group HTs, and the hierarchical Bloom filter BF are stored in a storage device such as the ROM 102, the RAM 103, and the magnetic disk 105 shown in FIG. In FIG. 2, the data block set db, the hash table group HTs, and the hierarchical Bloom filter BF are stored in the management device 200, but may be stored in an external device outside the management device 200. In this case, the management apparatus 200 reads out from or writes to the external apparatus via the network.

  When the data to be registered is entered in the data block set db, the registration processing unit 201 registers in the free area of the data block set db. When registering data, a hash value is obtained by a hash function, and the data (or its pointer) and the hash value are added to the hash table HT # corresponding to the data block db # of the registration destination. In addition, the registration processing unit 201 updates the hierarchical Bloom filter BF so that the hierarchical Bloom filter BF learns that data has been newly registered in the registration destination data block db #.

  When the search target data is input, the search processing unit 202 refers to the hierarchical Bloom filter BF and specifies the data block db # in which the search target data will exist. When the data block db # that may contain the search target data is not specified, the search target data does not exist in any data block db # (negative). On the other hand, even when the data block db # that may contain the search target data is specified, the specified data block db # does not necessarily include the search target data (false positive).

  Whether the false positive becomes positive or negative depends on the search result in the hash table HT # corresponding to the data block db # finally specified by the search processing unit 202. For example, in the hash table HT # corresponding to the data block db # finally specified by the search processing unit 202, it is positive if the hash value of the search target data is hit, and is negative if it is not hit.

  Specifically, the above-described registration processing unit 201 and search processing unit 202 execute, for example, the CPU 101 on a program stored in a storage device such as the ROM 102, the RAM 103, the magnetic disk 105, and the optical disk 107 illustrated in FIG. To realize its function.

<Hierarchical Bloom Filter BF>
FIG. 4 is an explanatory diagram showing an example of the hierarchical Bloom filter BF. The hierarchical Bloom filter BF is configured by a memory area of h stages × s bits width. The s bit width corresponds to the bit width of the data block set db. The bit length s of each stage is divided based on the division number d of the h-th stage that is the uppermost stage. Each of the divided parts is a Bloom filter and forms a Bloom filter row in each stage. The number of divisions d is basically an integer of 2 or more, but d = 1 may be used when the h-th stage as the uppermost stage is a single Bloom filter.

When an arbitrary stage is p, the bit width m of the Bloom filter bf (p) constituting the p-th Bloom filter row BF (p) is m = s / d ^{[h− (p−1)].} . In FIG. 4, d = 2. The number n of arrangement of the Bloom filters bf (p) in the p-th Bloom filter row BF (p) is n = d ^{[h− (p−1)]} .

  Therefore, in the hierarchical Bloom filter BF, the number of arrangements of the Bloom filters bf (p) in the p-th Bloom filter row BF (p) increases as the level decreases (h decreases). Note that the number of arrangement of the bloom filters bf (1) in the lowermost (first) Bloom filter row Bf (1) is the same as the number of data blocks db #.

  As a result, the Bloom filter bf (1) bitted when reaching the first level has a one-to-one correspondence with the data block db #. The number of stages h of the hierarchical Bloom filter BF is basically a plurality of stages, but may be one stage (h = 1). However, in this case, d ≠ 1.

<Example of learning process of hierarchical Bloom filter BF>
FIG. 5 is an explanatory diagram showing an example of learning processing of the hierarchical Bloom filter BF by the registration processing unit 201. In FIG. 5, for the sake of explanation, it is assumed that the total bit width s = 4096 bits, the number of stages h = 3, and the number of divisions d = 2 at the h-th stage.

Accordingly, the first-stage Bloom filter row BF (1), which is the lowermost stage, is divided into 8 (= d ^{[h- (p-1)]} = 2 ³ ) and Bloom filters bf (1-1) to bf ( 1-8). The second-stage Bloom filter row BF (2) is divided into 4 (= d ^{[h- (p-1)]} = 2 ² ) to obtain Bloom filters bf (2-1) to bf (2-4). Consists of. The third-stage Bloom filter row BF (3), which is the uppermost stage, is divided into 2 (= d ^{[h- (p-1)]} = 2 ¹ ) and Bloom filters bf (3-1) to bf ( 3-2).

  Further, the number k of hash functions to which the registered target data D is given is set to k = 3. Here, the hash functions H1 (), H2 (), and H3 () are used. A hash function to be registered in the hash table is H1 ().

First, it is assumed that the target data D is registered in the data block db3 in the data block set db. As an example, the hash values when the target data D is given to the hash functions H1 (), H2 (), and H3 () are as follows.
H1 (D) = 1234567
H2 (D) = 3984012
H3 (D) = 9803323

  In the learning process of the hierarchical Bloom filter BF, a specific bit in the Bloom filter to be updated is turned on, but if the specific bit is already turned on, it is left as it is.

  Here, the registration processing unit 201 creates a hash table entry E3 for the hash table HT3 of the block number 3 of the data block db3 of the registration destination. Then, the registration processing unit 201 additionally registers the created hash table entry E3 in the hash table HT3.

  Next, the registration processing unit 201 specifies a Bloom filter to be updated from the first-stage Bloom filter row BF (1). In the bottom row, the Bloom filter bf (1-3) having the same array number as the block number 3 of the data block db3 of the registration destination corresponds. Therefore, the Bloom filter bf (1-3) is the update target. The Bloom filter bf (1-3) is a 512-bit bit string.

  Also, the registration processing unit 201 divides each hash value by 512 that is the bit width of the first-stage Bloom filter bf (1), and calculates a remainder value. It is assumed that the remainder value of the hash value H1 (D) is “135”, the remainder value of the hash value H2 (D) is “140”, and the remainder value of the hash value H3 (D) is “59”.

  Next, the registration processing unit 201 turns on the bit at the position corresponding to the remainder value in the Bloom filter to be updated. If the remainder value is 0, the last bit of the Bloom filter to be updated is turned ON. In the example of FIG. 5, the Bloom filter bf (1-3) has 512 bits, so the 135th bit from the beginning is turned ON for the remainder value “135”. Similarly, for the remainder value “140”, the 140th bit from the beginning is turned ON. For the remainder value “59”, the 59th bit from the beginning is turned ON. This completes the learning process in the first stage.

  Next, the learning process of the second stage is started. The registration processing unit 201 identifies the Bloom filter to be updated from the second-stage Bloom filter row BF (2). Specifically, the Bloom filter including the bit position of the Bloom filter bf (1-3) to be updated in the first stage is specified from the Bloom filter string BF (2) in the second stage. In this example, the Bloom filter bf (2-2) is obtained. More specifically, the array number 3 of the Bloom filter bf (1-3) to be updated in the first stage of the previous stage is divided by the division number d (= 2), and the fraction is rounded up to round up the update target. The sequence number is 2. Therefore, the Bloom filter bf (2-2) is specified.

  Then, the registration processing unit 201 divides each hash value by 1024 which is the bit width of the second-stage Bloom filter bf (2), and calculates a remainder value. It is assumed that the remainder value of the hash value H1 (D) is “647”, the remainder value of the hash value H2 (D) is “652”, and the remainder value of the hash value H3 (D) is “571”.

  Next, the registration processing unit 201 turns on the bit at the position corresponding to the remainder value in the Bloom filter to be updated. If the remainder value is 0, the last bit of the Bloom filter to be updated is turned ON. In the example of FIG. 5, the Bloom filter bf (2-2) has 1024 bits, so the 647th bit from the beginning is turned ON for the remainder value “647”. Similarly, for the remainder value “652,” the 652nd bit from the beginning is turned ON. For the remainder value “571”, the 571st bit from the beginning is turned ON. Thereby, the learning process in the second stage is completed.

  Next, the process moves to the third-stage learning process, which is the uppermost stage. The registration processing unit 201 specifies the Bloom filter to be updated from the third-stage Bloom filter row BF (3). Specifically, the Bloom filter including the bit position of the Bloom filter bf (2-2) to be updated in the second stage is specified from the third stage Bloom filter string BF (3). In this example, the Bloom filter bf (3-1) is obtained. More specifically, by dividing the array number 2 of the Bloom filter bf (2-2) to be updated in the second stage of the previous stage by the division number d (= 2), the array number to be updated is 1 It becomes. Therefore, the Bloom filter bf (3-1) is specified.

  Then, the registration processing unit 201 divides each hash value by 2048 which is the bit width of the third-stage Bloom filter bf (3) to calculate a remainder value. It is assumed that the remainder value of the hash value H1 (D) is “1671”, the remainder value of the hash value H2 (D) is “652”, and the remainder value of the hash value H3 (D) is “1595”.

  Next, the registration processing unit 201 turns on the bit at the position corresponding to the remainder value in the Bloom filter to be updated. If the remainder value is 0, the last bit of the Bloom filter to be updated is turned ON. In the example of FIG. 5, the Bloom filter bf (3-1) has 2048 bits, so the 1671th bit from the beginning is turned ON for the remainder value “1671”. Similarly, for the remainder value “652,” the 652nd bit from the beginning is turned ON. For the remainder value “1595”, the 1595th bit from the beginning is turned ON. Thereby, the learning process in the third stage is finished.

  With this procedure, the registration processing unit 201 causes the hierarchical Bloom filter BF to learn data entries.

<Example of search processing using hierarchical Bloom filter BF>
FIG. 6 is an explanatory diagram showing an example of search processing of the hierarchical Bloom filter BF by the search processing unit 202. 6 will be described using the same hierarchical Bloom filter BF as in FIG. Note that FIG. 6 illustrates an example in which the target data D registered in FIG. 5 is searched as search target data.

  In the learning process shown in FIG. 5, processing is performed from the first stage at the bottom, but in the search process, processing is performed from the top (the third stage in the case of FIG. 6). First, the search processing unit 202 divides the three hash values for the target data D by the bit widths 2048 of the third-stage Bloom filters bf (3), resulting in residual values “1671”, “652”, “1595” is obtained.

  Next, the search processing unit 202 specifies the Bloom filter to be filtered from the third-stage Bloom filter row BF (3). Since the third stage is the top stage, all the Bloom filters bf (3-1) and bf (3-2) in the third stage are specified unconditionally.

  Then, the search processing unit 202 specifies a Bloom filter in which all the bits at the position corresponding to the calculated remainder value are ON from the Bloom filter group specified as the filtering target. In the case of the third stage, it is assumed that all of the bits at the position corresponding to the calculated remainder value are ON in the Bloom filters bf (3-1) and bf (3-2). Thereby, the filtering process in the third stage is finished.

  Next, the process proceeds to the second-stage filtering process. The search processing unit 202 divides the three hash values of the target data D by the bit width 1024 of each second-stage Bloom filter bf (2), and the remainder values “647”, “652”, “571 "

  Next, the search processing unit 202 specifies the Bloom filter to be filtered from the second-stage Bloom filter row BF (2). In the case of the remaining stages excluding the uppermost stage, the Bloom filter bf (p + 1) in which all the bits in the position corresponding to the calculated remainder value are turned on is searched in the upper stage, and the Bloom filter bf (p + 1). The Bloom filter bf (p) included in the bit positions of is the filtering target.

  In the case of the second stage, the blooms included in the bit positions of the Bloom filters bf (3-1) and bf (3-2) in which all the bits at the positions corresponding to the remainder values calculated in the third stage are ON. Filters bf (2-1) to bf (2-4) are targeted for filtering.

  Then, the search processing unit 202 specifies a Bloom filter in which all the bits at the position corresponding to the calculated remainder value are ON from the Bloom filter group specified as the filtering target. In the case of the second stage, the Bloom filters bf (2-2) and bf (2-3) are assumed to have all the bits at the positions corresponding to the calculated remainder values turned on. On the other hand, for the Bloom filters bf (2-1) and bf (2-4), it is assumed that one of the bits at the position corresponding to the calculated remainder value is OFF.

  Therefore, the lower-stage Bloom filters bf (1-1), bf (1-2), bf (1-7) included in the bit positions of the Bloom filters bf (2-1) and bf (2-4). , Bf (1-8) are excluded from filtering, and the data block db # in which the target data D exists is a data block included in the bit positions of the Bloom filters bf (2-2), bf (2-3). It is narrowed down to db #. Thereby, the filtering process in the second stage is completed.

  Next, the process proceeds to the first-stage filtering process which is the lowest stage. The search processing unit 202 divides the three hash values for the target data D by the bit width 512 of each Bloom filter bf (1) in the first stage, resulting in residual values “135”, “140”, “59 "

  Next, the search processing unit 202 specifies a Bloom filter to be filtered from the first-stage Bloom filter row BF (1). In the case of the first stage, the bloom included in the bit positions of the Bloom filters bf (2-2) and bf (2-3) in which all the bits at the positions corresponding to the remainder values calculated in the second stage are ON. Filters bf (1-3) to bf (1-6) are targeted for filtering.

  Then, the search processing unit 202 specifies a Bloom filter in which all the bits at the position corresponding to the calculated remainder value are ON from the Bloom filter group specified as the filtering target. In the case of the first stage, the Bloom filters bf (1-3) and bf (1-6) are assumed to have all the bits at positions corresponding to the calculated remainder values turned ON. On the other hand, for the Bloom filters bf (1-4) and bf (1-5), it is assumed that one of the bits at the position corresponding to the calculated remainder value is OFF.

  At the bottom level, there are no further lower levels, so the Bloom filters bf (1-3) and bf (1-6) have false positives. The search processing unit 202 determines whether or not the hash value H1 (D) is registered in the hash table HT3 corresponding to the array number “3” of the identified Bloom filter bf (1-3). Since the entry E3 is registered in the hash table HT3, it is found that the target data D is registered in the data block db3 corresponding to the hash table HT3.

  On the other hand, the search processing unit 202 also determines whether or not the hash value H1 (D) is registered in the hash table HT6 corresponding to the array number “6” of the identified Bloom filter bf (1-6). In the hash table HT6, since the hash value H1 (D) = 1234567 is not registered, it is found that the target data D is not registered in the data block db6 corresponding to the hash table HT6. Thereby, the search process is terminated.

  By such a procedure, the search processing unit 202 can specify the data block in which the target data D exists using the hierarchical Bloom filter BF.

  Next, the influence of the false positive of the Bloom filter will be described.

  The Bloom filter false-positive occurrence probability FPR is as follows when the number of data registrations is N (N <m) and the number of hash functions is k when there are h Bloom filters with a bit length of m. It can be expressed as equation (2).

FPR = {1- (1-1 / m ) kN} k ≒ {1-e (-kN / m))} k ... (1)

  In this case, by changing k, m, and N, the false positive occurrence probability FPR can be made very small. That is, in the present embodiment, the false positive occurrence probability FPR can be set to a value (substantially 0) that is much smaller than 1 depending on the setting of k, m, and N. Therefore, in the example of FIG. 5, the Bloom filter bf (1-6) is almost never selected.

In the present embodiment, since the data block number Ndb is d ^h , the height step number h can be expressed by the following equation (2).

  h = log (Ndb) / log (d) +1 (2)

  The above formula (2) is based on the assumption that log (Ndb) / log (d) is divisible, but if not, h can be determined by changing the value of d from other stages depending on the stage. it can.

  Further, in the above-described search process, it is necessary to perform collation as many times as the number of hash values (k times (constant)), and the number of filtering targets per search is at most d. Therefore, the memory access count MA by search is at most about the level expressed by the following equation (3).

  MA = k * d * log (Ndb) / log (d) (3)

  That is, the number of stages h (= memory amount) can be reduced by increasing the number of divisions d, while the number of searches is in a trade-off relationship that increases as the number of divisions d increases. Therefore, by considering this relationship, an appropriate memory operation can be performed.

<Hierarchical transposed Bloom filter>
Next, the hierarchical transposed Bloom filter will be described. In the above description, the registration process and the search process of the hierarchical Bloom filter BF have been described. However, the hierarchical Bloom filter BF is transposed in order to increase the search speed.

  FIG. 7 is an explanatory diagram showing a transposition example of the p-th Bloom filter row bf (p) in the hierarchical Bloom filter BF. (A) shows the Bloom filter row BF (p). Here, the Bloom filter row BF (p) shows Bloom filters bf (p−1) to bf (p−4) divided into four as an example. That is, the Bloom filter string BF (p) is a bit string of 10 bits × 4 filters. In the case of transposition, the bit string is 4 bits × 10 filters.

  (B) shows transposition of the Bloom filter row BF (p). When transposing, the bits at the same position of each of the Bloom filters bf (p-1) to bf (p-4) are collected, and the bit strings collected at the same positions are arranged in the order of the bit positions.

  Specifically, the first bits of each of the Bloom filters bf (p-1) to bf (p-4) are grouped in the order of the array number to form a bit string {0110}. From the left, the first bit “0” is the first bit of the Bloom filter bf (p−1), the second bit “1” is the first bit of the Bloom filter bf (p−2), and the third bit “1” is the Bloom filter The leading bit and trailing bit “0” of bf (p−3) are the leading bit of the Bloom filter bf (p−4).

  This bit string {0110} is referred to as a transposed Bloom filter tbf (p−1). The transposed Bloom filters tbf (p-1) to tbf (p-10) are obtained by collecting the second to last bit positions in the same manner. The index information in which the transposed Bloom filters tbf (p-1) to tbf (p-10) are arranged in the order of bit positions is referred to as a transposed Bloom filter row tBF (p). By generating the transposed Bloom filter row tBF (p) at all stages, the hierarchical transposed Bloom filter tBF is obtained.

  (C) shows a search comparison example between the Bloom filter row BF (p) and the transposed Bloom filter row tBF (p). Here, two hash values of the target data D are obtained by two types of hash functions, and a remainder value obtained by dividing by the bit width 10 of the Bloom filter bf (p) constituting the Bloom filter string BF (p) is “4”. And “8”.

  When searching with the Bloom filter column BF (p), the Bloom filter bf (p) in which the bit positions “4” and “8” having the remainder values “4” and “8” are all ON is selected as the Bloom filter column BF. Search from (p). In this case, the Bloom filter bf (p-2) is applicable.

  On the other hand, when the transposed Bloom filter row tBF (p) is used, the Bloom filter bf (p) in which the bit positions “4” and “8” are all ON, such as the Bloom filter row BF (p), is searched. First, transposed Bloom filters tbf (p-4) and tbf (p-8) having the same sequence numbers as the remainder values “4” and “8” are extracted. Then, an AND operation is performed on the extracted transposed Bloom filters tbf (p−4) and tbf (p−8) to identify the bit position “2” that is both ON.

  In the case of the Bloom filter string BF (p), since the fourth and eighth bits in the four Bloom filters bf (p-1) to bf (p-4) are referred to, 8 (= 4 × 2) Memory access is required. On the other hand, the transposed Bloom filter row tBF (p) is index information folded for each bit position of the Bloom filters bf (p-1) to bf (p-4) before transposition. Therefore, it is possible to make a determination by two memory accesses of extracting the transposed Bloom filters tbf (p-4) and tbf (p-8) and their AND operation. Therefore, the memory access frequency is reduced, and the search speed is increased.

<Example of search processing using hierarchical transposed Bloom filter>
FIG. 8 is an explanatory diagram showing a search process example of the hierarchical transposed Bloom filter by the search processing unit 202. In FIG. 8, for the sake of explanation, the total bit width s = 64 bits, the number of stages h = 3, and the number of divisions d = 2 at the h-th stage.

Since the bit width of the Bloom filter constituting the first-stage Bloom filter column BF (1) which is the lowest stage is 8 (= s / d ^h = 64/2 ³ ) bits, the first stage which is the lowest stage The transposed Bloom filter row tBF (1) includes 8 (= s / d ^h = 64/2 ³ ) transposed Bloom filters tbf (1-1) to tbf (1-8).

Further, since the bit width of the Bloom filter constituting the second-stage transposed Bloom filter string tBF (2) is 16 (= s / d ^h = 64/2 ² ) bits, the second-stage transposed Bloom filter string tBF (2) is composed of 16 (= s / d ^h = 64/2 ² ) transposed Bloom filters tbf (2-1) to tbf (2-16).

The bit width of the Bloom filter constituting the third-stage Bloom filter column BF (3), which is the uppermost stage, is 32 (= s / d ^h = 64/2 ¹ ) bits, so The three-stage transposed Bloom filter row tBF (3) includes 32 (= s / d ^h = 64/2 ¹ ) transposed Bloom filters tbf (3-1) to tbf (3-32).

  In FIG. 8, for the purpose of explanation, the transposed Bloom filter rows tBF (1) to tBF (3) and the pre-transposed Bloom filter rows BF (1) to BF (3) are also shown for comparison.

  First, the search processing unit 202 divides the three hash values of the hash functions H1 () to H3 () for the search target data Dx by the number of transposed Bloom filters 32 in the third stage “2” ”,“ 19 ”,“ 27 ”.

  Next, the search processing unit 202 identifies the transposed Bloom filter to be filtered from the third-stage transposed Bloom filter string tBF (3). Specifically, the transposed Bloom filters tbf (3-2), tbf (3-19), and tbf (3-27) that are in the same bit position as the remainder value (or the end position when the remainder value is 0) are specified. . Then, an AND operation is performed on each bit string {10}, {11}, {10} of the identified transposed Bloom filter tbf (3-2), tbf (3-19), tbf (3-27). The AND result is {10}.

  When “1” is not included in the AND result, the search processing unit 202 determines that the search target data Dx does not exist in the data block set db. On the other hand, when “1” is included in the AND result, the search target data Dx may be registered, and the process moves to the next lower level.

  Also in the second stage, first, the search processing unit 202 divides the three hash values for the search target data Dx by the number of transposed Bloom filters 16 in the second stage, “8” and “11”. , “13” is obtained.

  Next, the search processing unit 202 specifies the transposed Bloom filter to be filtered from the second-stage transposed Bloom filter row tBF (2). Specifically, the transposed Bloom filters tbf (2-8), tbf (2-11), and tbf (2-13) that have the same bit position as the remainder value (or the end position when the remainder value is 0) are specified. . Then, an AND operation is performed on each bit string {0110}, {0100}, {0110} of the identified transposed Bloom filters tbf (2-8), tbf (2-11), and tbf (2-13). The AND result is {0100}.

  When “1” is not included in the AND result, the search processing unit 202 determines that the search target data Dx does not exist in the data block set db. On the other hand, when “1” is included in the AND result, the search target data Dx may be registered, and the process moves to the next lower level.

  Even in the first level, which is the lowest level, first, the search processing unit 202 divides the three hash values for the search target data Dx by the number of transposed Bloom filters of the first level “8”. , “5”, “7” are obtained.

  Next, the search processing unit 202 identifies the transposed Bloom filter to be filtered from the first-stage transposed Bloom filter string tBF (1). Specifically, the transposed Bloom filters tbf (1-2), tbf (1-5), and tbf (1-7) that are in the same bit position as the remainder value (or the end position when the remainder value is 0) are specified. . Then, an AND operation is performed on each bit string {00110110}, {10011010}, and {00110111} of the identified transposed Bloom filters tbf (1-2), tbf (1-5), and tbf (1-7). The AND result is {00010010}.

  Since there is no lower stage, there is a possibility that the search target data Dx exists in the data blocks db4 and db7 corresponding to the bit positions 4 and 7 whose AND result {00010010} is “1” due to false positive.

  In this case, it is assumed that the data block db4 is hit and the data block db7 is not hit by searching the hash tables HT4 and HT7 using the hash value of the hash function H1 () as a key. Thereby, it is found that the search target data Dx is registered in the data block db4. Thereby, the search process is terminated.

  By such a procedure, the search processing unit 202 can search for target data at a higher speed than the hierarchical Bloom filter BF by using the hierarchical transposed Bloom filter.

<Detailed Functional Configuration of Search Processing Unit 202>
Next, a detailed functional configuration example of the search processing unit 202 will be described.

  FIG. 9 is a block diagram illustrating a functional configuration example of the search processing unit 202. The search processing unit 202 includes a receiving unit 901, a transposing unit 902, a converting unit 903, a first specifying unit 904, a second specifying unit 905, a determining unit 906, a determining unit 907, and an extracting unit 908. And an output unit 909.

  The receiving unit 901 has a function of receiving a transposition request for the Bloom filter row BF (p). For example, a transposition request from the hierarchical Bloom filter BF to the hierarchical transposed Bloom filter tBF is accepted.

  Here, the transposition request is a request for transposing the Bloom filter string BF (p) to the transposed Bloom filter string tBF (p). For example, the activation completion notification that the management apparatus 200 is activated and the activation is completed is used as the transposition request. Thereby, when the management apparatus 200 is activated, the hierarchical Bloom filter BF is transposed to the hierarchical transposed Bloom filter tBF. In this way, if the entire transposed Bloom filter tBF is transposed at the time of activation, the transposed Bloom filter tBF can be used at all times until shutdown.

  The search request for the data block set db may be a transposition request. Thereby, the search processing unit 202 stands by until a search request is made, and when there is a search request, the Bloom filter string BF (p) used in the search processing unit 202 is transposed among the hierarchical Bloom filters BF. It will be. Thereby, the transposed Bloom filter string tBF (p) increases as the search frequency increases. Therefore, since it is gradually transposed from the Bloom filter row BF (p) necessary for the search, it is possible to reduce useless transposition processing of the Bloom filter row that was not used until the shutdown.

  The transposition unit 902 has a function of transposing the Bloom filter string BF (p) to the transposed Bloom filter string tBF (p) when the transposition request is accepted by the accepting unit 901. Specifically, the transposition unit 902 transposes the Bloom filter row BF (p) to the transposed Bloom filter row tBF (p) by the method shown in FIG.

For example, the Bloom filter row BF (p) is index information in which n (= d ^{[h− (p−1)]} ) Bloom filters bf (p−1) to bf (pn) are arranged. . The bit width of each of the Bloom filters bf (p−1) to bf (p−n) is m (= s / n) bits.

When the Bloom filter row BF (p) is transposed according to FIG. 7, the transposed Bloom filter row tBF (p) has m (= s / n) transposed Bloom filters tbf (p−1) to tbf (p−m). Becomes the index information arranged. The bit width of each transposed Bloom filter tbf (p-1) to tbf (pm) is n (= d ^{[h- (p-1)]} ) bits. In this way, the number of arrays and the bit width are interchanged by transposition.

  The conversion unit 903 has a function of converting the search target data into position information representing the array position of the transposed Bloom filter for each hash function based on a plurality of types (number of types k) of hash functions. Specifically, for example, k hash values are obtained by providing search target data to k different hash functions H1 () to Hk ().

  In the entry to the Bloom filter column BF (p) shown in FIG. 5, the hash value is divided by the bit width m (= s / n) of the Bloom filter constituting the Bloom filter column BF (p), and the remainder is obtained. The corresponding bit position was set to ON.

  Therefore, when k hash values are obtained, the conversion unit 903 divides the hash value by the number m (= s / n) of transposed Bloom filter arrays constituting the transposed Bloom filter sequence tBF (p), Calculate the remainder value. The k remainder values are position information representing the arrangement position of the transposed Bloom filter. Note that the position information when the calculated remainder value is 0 is m.

  The first specifying unit 904 has a function of specifying, for each positional information, a transposed Bloom filter corresponding to the positional information converted by the converting unit 903 from the transposed Bloom filter row tBF (p). Specifically, a transposed Bloom filter whose array number matches the position information obtained from the conversion unit 903 is specified from the transposed Bloom filter row tBF (p).

  In the example illustrated in FIG. 8, in the third-stage transposed Bloom filter row tBF (3), the remainder values (position information) are “2”, “19”, and “27”, so the array element number is “2”. , “19”, “27” are specified as transposed Bloom filters tbf (3-2), tbf (3-19), tbf (3-27).

  The second specifying unit 905 generates the Bloom filter bf (p) corresponding to the position information common to the plurality of transposed Bloom filters tbf (p) specified by the first specifying unit 904, and the Bloom filter row BF (p). It has a function to identify from. Specifically, an AND operation is performed on the bit strings of the plurality of transposed Bloom filters tbf (p) specified by the first specifying unit 904. The bit position where “1” appears by the AND operation is a Bloom filter having a bit that would have been turned ON at the time of entry in the Bloom filter string BF (p) before transposition.

  In the example shown in FIG. 8, when the transposed Bloom filters tbf (3-2), tbf (3-19), and tbf (3-27) are ANDed in the third stage, the leading bit position in the AND result is “1”. " Therefore, it can be seen that any bit of the Bloom filter bf (3-1) in the Bloom filter row BF (3) before transposition may have been turned ON at the time of entry of search target data.

  As described above, the first specifying unit 904 and the second specifying unit 905 reduce the number of accesses to the transposed Bloom filter row tBF (p) than that before the transposed Bloom filter row BF (p). Therefore, as the number of stages p of the transposed Bloom filter tbf (p) increases, the number of accesses is significantly suppressed compared to the pre-transposed Bloom filter string BF (p), and the search speed can be increased. .

  The determination unit 906 has a function of determining whether there is a Bloom filter row having the Bloom filter specified by the second specifying unit 905 as an array element. Specifically, it is only necessary to determine whether or not there is a transposed Bloom filter row one level below the transposed Bloom filter row tBF (p). More specifically, it is only necessary to check whether p = 1.

  When the transposed Bloom filter row of the next lower stage exists (p ≠ 1), the first specifying unit 904 newly converts the transposed Bloom filter row of the lower stage to the transposed Bloom filter row tBF (p). And On the other hand, when there is no lower-stage transposed Bloom filter row (p = 1), the transposed Bloom filter row tBF (p) becomes the first-stage transposed Bloom filter row tBF (1), which is the lowest stage. .

  If the determination unit 906 determines that the data does not exist, the determination unit 907 includes the data block db # corresponding to the Bloom filter bf (p) specified by the second specification unit 905 in the data block set db. It has a function of determining the presence or absence of search target data. When p = 1, since the Bloom filter bf (1) specified by the second specifying unit 905 has a one-to-one correspondence with the data block, the bit that is “1” in the AND result of the second specifying unit 905 The data block db # having the block number # as the position is determined as a determination target. The determination unit 907 determines whether the determination target data block db # is positive or negative and therefore is positive or negative.

  Then, with reference to the hash table HT # of the determination target data block db #, false positive / negative of the search target data is determined. In this case, similarly to the search process using the hierarchical Bloom filter BF, data (or a pointer thereof) from the hash table HT # using a hash value of search target data by a specific hash function (for example, H1 ()) as a key. The presence or absence of is determined. If it does not exist, it means that a miss has occurred due to a false positive.

  The extraction unit 908 has a function of extracting search target data from the determination target data block db # when the determination unit 907 determines that the data is positive. Specifically, for example, since the data storage position is determined by the hash table HT # corresponding to the determination target data block, the extraction unit 908 obtains the search target data and the data related to the search target data from the storage position. Extract.

  For example, the search target data itself is extracted from the determination target data block db #. If it can be extracted, it is found that the search target data is registered. If the search target data is a file number, file data related to the file number is extracted. In addition, when the search target data is a dictionary or a term headword, the term explanation data of the headword is extracted.

  The output unit 909 outputs the determination result by the determination unit 907 and the data extracted by the extraction unit 908. For example, the determination result such as positive or negative of the search target data and the extracted data when it is positive are output. Examples of the output format include display on the display 108, audio output, print output, transmission to another apparatus, and the like.

<Learning Processing Procedure of Hierarchical Bloom Filter BF by Registration Processing Unit 201>
FIG. 10 is a flowchart showing the learning processing procedure of the hierarchical Bloom filter BF by the registration processing unit 201. First, the registration processing unit 201 determines whether there is data to be registered (target data D) (step S1001). When there is target data D (step S1001: Yes), the registration processing unit 201 sets the number of stages p to p = 1 (step S1002), and p> h (h is the uppermost stage of the hierarchical Bloom filter BF). Whether or not (step S1003). If p> h is not satisfied (step S1003: NO), the registration processing unit 201 calculates k hash values of the target data using k types of hash functions (step S1004). Then, k hash values are divided by the bit width of the Bloom filter string BF (p) to calculate k remainder values (step S1005).

  Next, the registration processing unit 201 identifies the bloom filter bf (p) r that is the registration destination from the p-th Bloom filter row BF (p) (step S1006). When p = 1, the bloom filter bf (1- #) corresponding to the block number # of the data block db # of the storage destination is set as the bloom filter bf (p) r of the registration destination.

  Specifically, as shown in FIG. 5, when registering the target data D in the data block db3, the first-stage Bloom filter bf (1-3) having the same array number as the block number 3 of the data block db3. ) Is the bloom filter bf (1) r of the registration destination.

  If p ≠ 1, the Bloom filter corresponding to the bit position of the (p−1) -th registered Bloom filter bf (p−1) r from the p-th Bloom filter string BF (p) Let bf (p) be the newly registered Bloom filter bf (p) r.

  Specifically, as shown in FIG. 5, when P = 2, the Bloom filter bf (1) r of the first registration destination is selected from the second-stage Bloom filter row BF (2). The Bloom filter bf (2-2) including the bit position of the Bloom filter bf (1-3) is set as a new registration destination Bloom filter bf (2) r.

  Then, the k remainder values calculated in step S1005 are entered in the registered Bloom filter bf (p) r (step S1007). That is, the bit at the same bit position as the remainder value is turned ON. When the remainder value is 0, the tail bit is turned ON. Then, the stage number p is incremented (step S1008), and the process returns to step S1003.

  If p> h in step S1003 (step S1003: Yes), a hash table entry of the target data is added (step S1009). Specifically, for example, as shown in FIG. 4, the hash table entry E3 is additionally registered in the hash table HT3.

  Then, the process returns to step S1001. In step S1001, if there is no target data D (step S1001: No), the learning process of the hierarchical Bloom filter BF by the registration processing unit 201 is terminated. By such a processing procedure, the hierarchical Bloom filter BF is constructed.

<Search Processing Procedure by Search Processing Unit 202>
FIG. 11 and FIG. 12 are flowcharts showing a search processing procedure by the search processing unit 202. Note that the Bloom filter used by the search processing unit 202 may be a hierarchical transposed Bloom filter tBF that has already been transposed, or a hierarchical Bloom filter BF. When the hierarchical Bloom filter BF is used, a transposition process is performed each time an untransposed Bloom filter is specified.

  That is, the hierarchical transposed Bloom filter tBF is constructed while searching. 11 and 12, the repair procedure when the hierarchical transposed Bloom filter tBF is constructed while searching will be described. In addition, when batch transposition has been completed, it is only necessary to omit the determination of whether transposition has been performed (step S1104) and the transposition processing (step S1105).

  First, in FIG. 11, the search processing unit 202 waits for the search target data Dx by the receiving unit 901 (step S1101: No), and when the search target data Dx is received (step S1101: Yes), the search processing unit 202. The conversion unit 903 gives the search object data Dx to k types of hash functions to calculate k hash values (step S1102).

  Then, the search processing unit 202 sets p = h, that is, sets the number of stages p to the maximum number of stages h (step S1103), and determines whether or not the p-th Bloom filter string BF (p) has been transposed. (Step S1104). When it has been transposed (step S1104: Yes), the process proceeds to step S1106. On the other hand, if it is not transposed (step S1104: No), the search processing unit 202 transposes the p-th Bloom filter row BF (p) by the transposing unit 902 (step S1105), and proceeds to step S1106. .

  In step S1106, the search processing unit 202 uses the conversion unit 903 to divide k hash values by the number of arrays of transposed Bloom filters to calculate k remainder values (step S1106). Then, the search processing unit 202 uses the first specifying unit 904 to extract k transposed Bloom filters tbf (p) r corresponding to k remainder values from the p-th transposed Bloom filter sequence tBF (p). Specify (step S1107).

  Then, the search processing unit 202 performs an AND operation on the k transposed Bloom filters tbf (p) r by the second specifying unit 905 (step S1108), and proceeds to step S1201 in FIG.

  In FIG. 12, the search processing unit 202 uses the second specifying unit 905 to set the first bit of the AND result as the target bit (step S1201), and determines whether the target bit is ON (step S1202). . If not ON, the search processing unit 202 uses the determination unit 907 to determine whether the target bit can be shifted (step S1203). Specifically, it is determined whether or not the target bit is the tail bit.

  When the shift is possible (step S1203: Yes), the search processing unit 202 shifts the target bit toward the end of one bit (step S1204), and returns to step S1202. On the other hand, if the shift is not possible in step S1203 (step S1203: No), the search processing unit 202 determines the search result (negative) by the determination unit 907 and outputs the result from the output unit 909 (step S1205). Thereby, the processing procedure when the search result is negative is terminated.

  On the other hand, if the target bit is ON in step S1202 (step S1202: Yes), the search processing unit 202 determines whether the current stage number p is p = 1 by the determination unit 906 (step S1202). S1206). When p is not 1 (step S1206: No), p is decremented (step S1207), and the process returns to step S1104.

  On the other hand, if p = 1 (step S1206: Yes), the search processing unit 202 uses the determination unit 907 to search for a hash table corresponding to the bit position of the target bit (step S1208). Then, it is determined whether or not the search target data Dx exists (step S1209).

  When the search target data does not exist (step S1209: No), the process returns to step S1203 to determine whether the target bit can be shifted. On the other hand, when search target data exists (step S1209: Yes), the search processing unit 202 outputs a search result (positive) (step S1210). The search processing unit 202 extracts related data by the extraction unit 908 as necessary and outputs it as a search result. Thereby, the processing procedure when the search result is negative is terminated.

  As described above, according to the present embodiment, transposing the Bloom filter string BF (p) to the transposed Bloom filter string tBF (p) reduces the memory access and increases the search speed. There is an effect. In particular, by using the hierarchical transposed Bloom filter tBF, memory access at each stage is reduced, so that a search can be performed at a higher speed.

  Moreover, when the management apparatus 200 is activated by using the activation completion notification as a transposition request, the hierarchical Bloom filter BF is transposed to the hierarchical transposed Bloom filter tBF. In this way, if the transposition to the hierarchical transposed Bloom filter tBF is collectively performed at the time of startup, the hierarchical transposed Bloom filter tBF can be used at all times until the shutdown.

  Also, by making the search request for the data block set db a transposition request, when there is a search request, the Bloom filter string BF (p) used in the search processing unit 202 in the hierarchical Bloom filter BF is transposed. . Thereby, the transposed Bloom filter string tBF (p) increases as the search frequency increases. Therefore, since it is gradually transposed from the Bloom filter row BF (p) necessary for the search, it is possible to reduce useless transposition processing of the Bloom filter row that was not used until the shutdown.

  Further, in the case of the hierarchical transposed Bloom filter tBF transposed from the hierarchical Bloom filter BF, until the bottom level is reached, whether the AND result is ON or not is determined whether it is false positive or negative, and the AND result is If there is even one ON bit, it is determined as a false positive and the process proceeds to the next lower stage. Thus, by using the AND result, it can be easily determined whether or not the result is negative, and it can be identified early that the search target data does not exist.

  In addition, since it is possible to search if only the hierarchical transposed Bloom filter tBF is held in the memory, it is not necessary to hold the hierarchical Bloom filter BF. Thereby, memory saving can be achieved.

<Example of learning process of hierarchical transposed Bloom filter tBF>
FIG. 13 is an explanatory diagram illustrating a learning process example of the hierarchical transposed Bloom filter tBF by the registration processing unit 201. Here, the hierarchical transposed Bloom filter tBF shown in FIG. 8 will be described as an example. Further, FIG. 8 illustrates an example in which the search target data Dx is searched and hit, but in FIG. 13, the search target data Dx (referred to as target data Dx in FIG. 13) is still registered in the data block set db. It will be described as not being done.

  Further, the number k of hash functions to which the registered target data D is given is set to k = 3. Here, the hash functions H1 (), H2 (), and H3 () are used. A hash function to be registered in the hash table is H1 ().

First, it is assumed that the target data Dx is registered in the data block db4 in the data block set db. As an example, the hash value when the target data Dx is given to each hash function H1 (), H2 (), H3 () is as follows.
H1 (Dx) = x1
H2 (Dx) = x2
H3 (Dx) = x3

  Further, in the learning process of the hierarchical transposed Bloom filter tBF, a specific bit in the transposed Bloom filter tbf (p) to be updated is turned ON, but if the specific bit is already ON, it is left as it is. To do.

  Here, the registration processing unit 201 creates a hash table entry E4 for the hash table HT4 of the block number 4 of the registration destination data block db4. Then, the registration processing unit 201 additionally registers the created hash table entry E4 in the hash table HT4.

  Then, the process proceeds to the first stage learning process. The registration processing unit 201 identifies the transposed Bloom filter tbf (1) to be updated from the first-stage transposed Bloom filter row tBF (1). Specifically, the registration processing unit 201 divides each hash value x1 to x3 by 8, which is the number of arrays of the first-stage transposed Bloom filter string tBF (1), and calculates a remainder value. It is assumed that the remainder value of the hash value x1 is “2”, the remainder value of the hash value x2 is “5”, and the remainder value of the hash value x3 is “7”. Therefore, the transposed Bloom filter tbf (1) to be updated in the first stage is the transposed Bloom filter tbf (1-2), tbf (1-5), tbf (1-7).

  In the lowest level, the bit position corresponding to the block number 4 of the registration-destination data block db4 is set as the update target bit. Therefore, the fourth bit from the beginning of the transposed Bloom filter to be updated is turned ON. Thereby, the learning process of the first-stage transposed Bloom filter row tBF (1) is completed.

  Next, the learning process of the second stage is started. The registration processing unit 201 identifies the transposed Bloom filter tbf (2) to be updated from the second-stage transposed Bloom filter row tBF (2). Specifically, the registration processing unit 201 divides each hash value x1 to x3 by 16, which is the number of arrays of the second-stage transposed Bloom filter string tBF (2), and calculates a remainder value. It is assumed that the remainder value of the hash value x1 is “8”, the remainder value of the hash value x2 is “11”, and the remainder value of the hash value x3 is “13”. Therefore, the transposed Bloom filter tbf (2) to be updated in the second stage is the transposed Bloom filter tbf (2-8), tbf (2-11), tbf (2-13).

Next, the bit position of the bit in the transposed Bloom filter tbf (2-8), tbf (2-11), tbf (2-13) will be described. In the hierarchical Bloom filter BF before transposition, the number of divisions is d, and each Bloom filter row BF (p) is divided into n (= d ^{[h− (p−1)]} ). As a result, the bit width of each Bloom filter column BF (p) is m (= s / n) bits.

  For this reason, in the learning process of the hierarchical Bloom filter BF, the Bloom filter bf (p) including the bit position of the Bloom filter bf ((p-1)-#) to be updated in the (p-1) -th stage is used. The p-th Bloom filter row BF (p) is specified.

  For example, in the example of FIG. 5, in the second stage, the array number 3 of the Bloom filter bf (1-3) to be updated in the first stage of the previous stage is divided by the division number d (= 2), and the fraction is calculated. By rounding up, the array number to be updated becomes 2. Therefore, the Bloom filter bf (2-2) is specified.

  On the other hand, in the hierarchical transposed Bloom filter tBF, the array number n and the bit width m are interchanged, so the array of Bloom filters bf ((p-1)-#) to be updated in the (p-1) stage. Instead of the number #, the bit position to be updated at the (p−1) -th stage is divided by the division number d, and the fraction is rounded up.

  In the case of the second stage, the update target bit in the first stage is the fourth bit from the head, and the transposed Bloom filters tbf (1-2), tbf (1-5), tbf (1-7) The 4th bit was turned ON. Therefore, since the update target bit in the second stage is d = 2, the bit of 4 / d = 2 bit from the head is set as the update target bit. In this example, the second bit from the beginning of the transposed Bloom filters tbf (2-8), tbf (2-11), and tbf (2-13) is turned ON. Thereby, the learning process of the second-stage transposed Bloom filter row tBF (2) is completed.

  Next, the process moves to the third stage learning process. The registration processing unit 201 identifies the transposed Bloom filter tbf (3) to be updated from the third-stage transposed Bloom filter row tBF (3). Specifically, the registration processing unit 201 divides each hash value x1 to x3 by 32, which is the number of arrays of the third-stage transposed Bloom filter string tBF (3), and calculates a remainder value. It is assumed that the remainder value of the hash value x1 is “2”, the remainder value of the hash value x2 is “19”, and the remainder value of the hash value x3 is “27”. Therefore, the transposed Bloom filter tbf (3) to be updated in the third stage is the transposed Bloom filter tbf (3-2), tbf (3-19), tbf (3-27).

  Next, the update target bits in the transposed Bloom filters tbf (3-2), tbf (3-19), and tbf (3-27) are determined. Similar to the second stage, not the array number # of the Bloom filter bf ((p-1)-#) to be updated in the (p-1) stage, but the bit position to be updated in the (p-1) stage. Divide by the division number d and round up the fraction.

  In the case of the third stage, the update target bit in the second stage, which is the previous stage, is the second bit from the beginning, and is a transposed Bloom filter tbf (2-8), tbf (2-11), tbf (2- The second bit of 13) was turned ON. Accordingly, since the update target bit in the third stage is d = 2, the bit of 2 / d = 1 bit from the head is set as the update target bit. In this example, the first bit of the transposed Bloom filter tbf (3-2), tbf (3-19), tbf (3-27) is turned ON. Thereby, the learning process of the third-stage transposed Bloom filter row tBF (3) is completed.

<Learning Processing Procedure of Hierarchical Transposed Bloom Filter tBF by Registration Processing Unit 201>
FIG. 14 is a flowchart showing the learning processing procedure of the hierarchical Bloom filter BF by the registration processing unit 201. First, the registration processing unit 201 determines whether there is data to be registered (target data Dx) (step S1401). When there is target data Dx (step S1401: Yes), the registration processing unit 201 sets the number of stages p to p = 1 (step S1402), and p> h (h is the uppermost stage of the hierarchical transposed Bloom filter tBF). It is determined whether or not there is (step S1403). If p> h is not satisfied (step S1403: NO), the registration processing unit 201 calculates k hash values of the target data Dx using k types of hash functions (step S1404).

  Next, the registration processing unit 201 divides the k hash values by the number of arrays of the p-th transposed Bloom filter tBF (p) to calculate k remainder values (step S1405). Then, the registration processing unit 201 identifies k transposed Bloom filters tbf (p) r having the same array number as the k remainder values (step S1406).

  Thereafter, it is determined whether or not p = 1 (step S1407). If p = 1 (step S1407: Yes), the registration processing unit 201 determines k transposed Bloom filters tbf (p ) In r, the block number # of the data block db # to which the target data Dx belongs is entered (step S1408). That is, the block number # of the data block db # to which the target data Dx belongs is set as the update target bit position #, and the bit at the update target bit position # of the identified k transposed Bloom filters tbf (p) r is turned ON. To do. Then, control goes to a step S1410.

  On the other hand, if p ≠ 1 in step S1407 (step S1407: No), the k-th transposed Bloom filter tbf (p) r is updated in the (p−1) th stage of the target data Dx. A quotient (rounded up) obtained by dividing the bit position by the division number d is entered (step S1409). That is, the quotient (rounded up) obtained by dividing the update target bit position in the (p−1) -th stage of the target data Dx by the division number d is set as the update target bit position, and the identified k transposed Bloom filters tbf (P) The bit at the update target bit position of r is turned ON. Then, control goes to a step S1410.

  In step S1410, the registration processing unit 201 increments the number of stages p (step S1410) and returns to step S1403. Thereby, the update target bit can be turned ON from the lowest level to the highest level.

  On the other hand, in step S1403, when p> h (step S1403: Yes), the registration processing unit 201 adds a hash table entry of the target data (step S1411), and returns to step S1401. If there is no target data Dx (step S1401: No), the learning process of the hierarchical transposed Bloom filter tBF by the registration processing unit 201 is terminated.

  By such a procedure, the registration processing unit 201 causes the hierarchical transposed Bloom filter tBF to learn the data entry. That is, even when data is registered after transposition, it is not necessary to return from the hierarchical transposed Bloom filter tBF to the hierarchical Bloom filter BF, and learning can be performed with the transposed hierarchical transposed Bloom filter tBF. Therefore, useless processing such as returning before transposition is eliminated, and search efficiency can be improved.

  Note that the search method described in this embodiment can be realized by executing a program prepared in advance on a computer such as a personal computer or a workstation. The search program is recorded on a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, an MO, and a DVD, and is executed by being read from the recording medium by the computer. The search program may be distributed via a network such as the Internet.

200 management apparatus 201 registration processing unit 202 search processing unit 901 receiving unit 902 transposing unit 903 converting unit 904 first specifying unit 905 second specifying unit 906 determining unit 907 determining unit 908 extracting unit 909 output unit BF hierarchical Bloom filter db Data block set HTs Hash table group tBF Hierarchical transposed Bloom filter

Claims (8)

A data block set including a registered data group for each data block and n Bloom filters in which m bits indicating negative in a predetermined number of data blocks are arranged in units of the predetermined number of data blocks On the computer with access to the Bloom Filter column,
An accepting step for accepting a transposition request for the Bloom filter row;
When a transposition request is accepted by the accepting step, a transposed Bloom filter sequence in which m n-bit transposed Bloom filters are arranged in which the Bloom filter sequence is a group of bit sequences in each Bloom filter are arranged at the same position. A transposition step of transposing to a storage device,
Based on a plurality of types of hash functions, for each hash function, a conversion step of converting search target data into position information representing an array position of the transposed Bloom filter;
From the transposed Bloom filter row, a first identifying step that identifies, for each positional information, a transposed Bloom filter corresponding to the positional information converted by the converting step;
A determination step of determining presence / absence of the search target data in the data block set based on position information common to a plurality of transposed Bloom filters specified by the first specification step;
A search program characterized in that is executed. The reception process includes
The search program according to claim 1, wherein the computer startup completion notification is received as a transposition request for the Bloom filter row. The reception process includes
The search program according to claim 1, wherein a search request for the data block set is accepted as a transposition request for the Bloom filter string. Bloom filter corresponding to the position information common to a plurality of transposed Bloom filter identified by the previous SL first specific step, a second specifying step of specifying from the Bloom filter row,
Causing the computer to execute a determination step of determining whether or not there is a Bloom filter row having the Bloom filter specified by the second specification step as an array element,
The determination step includes
If it is determined that the data does not exist in the determination step, determining whether or not the search target data is present in the data block corresponding to the Bloom filter specified in the second specific step in the data block set. The search program according to any one of claims 1 to 3, characterized in that: A data block set including a registered data group for each data block and a bit string in n Bloom filters in which m bits indicating negative in a predetermined number of data blocks are arranged together by bits at the same position A computer having access to a transposed Bloom filter array in which m n-bit transposed Bloom filters are arranged;
Based on a plurality of types of hash functions, for each hash function, a conversion step of converting search target data into position information representing an array position of the transposed Bloom filter;
From the transposed Bloom filter row, a first identifying step that identifies, for each positional information, a transposed Bloom filter corresponding to the positional information converted by the converting step;
A second specifying step of specifying a Bloom filter corresponding to position information common to a plurality of transposed Bloom filters specified by the first specifying step from the Bloom filter row;
A determination step of determining whether there is a Bloom filter row having the Bloom filter specified by the second specification step as an array element;
A determination step of determining whether or not the search target data exists in the data block corresponding to the Bloom filter specified by the second specific step in the data block set when it is determined that the data does not exist by the determination step. When,
A search program characterized by causing a computer to execute. The first specific step includes
If it is determined by the determination step that the bloom filter column having the Bloom filter specified by the second specific step as an array element is set as the specific Bloom filter column, the transposition transposed from the specific Bloom filter column By specifying a Bloom filter string as a specific target transposed Bloom filter string, a specific target transposed Bloom filter corresponding to the positional information converted by the conversion step is specified for each positional information from the specific target transposed Bloom filter string And
The second specific step includes
The Bloom filter corresponding to the positional information common to the plurality of specific target transposed Bloom filters specified by the first specifying step is specified from the specific target Bloom filter row. Search program described in. A data block set including a registered data group for each data block and n Bloom filters in which m bits indicating negative in a predetermined number of data blocks are arranged in units of the predetermined number of data blocks A search device accessible to the Bloom filter column,
Receiving means for receiving a transposition request of the Bloom filter row;
When a transposition request is accepted by the accepting unit, a transposed Bloom filter sequence in which m n-bit transposed Bloom filters are arranged in which the Bloom filter sequence is a group of bit sequences in each Bloom filter. Transposing means for transposing to a storage device;
Based on a plurality of types of hash functions, for each hash function, conversion means for converting search target data into position information representing an array position of the transposed Bloom filter;
From the transposed Bloom filter row, a first identifying unit that identifies, for each positional information, a transposed Bloom filter corresponding to the positional information converted by the converting unit;
Determination means for determining presence or absence of the search target data in the data block set based on position information common to a plurality of transposed Bloom filters specified by the first specifying means;
A search device comprising: A data block set including a registered data group for each data block and n Bloom filters in which m bits indicating negative in a predetermined number of data blocks are arranged in units of the predetermined number of data blocks A computer with access to the Bloom Filter column,
An accepting step for accepting a transposition request for the Bloom filter row;
When a transposition request is accepted by the accepting step, a transposed Bloom filter sequence in which m n-bit transposed Bloom filters are arranged in which the Bloom filter sequence is a group of bit sequences in each Bloom filter are arranged at the same position. A transposition step of transposing to a storage device,
Based on a plurality of types of hash functions, for each hash function, a conversion step of converting search target data into position information representing an array position of the transposed Bloom filter;
From the transposed Bloom filter row, a first identifying step that identifies, for each positional information, a transposed Bloom filter corresponding to the positional information converted by the converting step;
A determination step of determining presence / absence of the search target data in the data block set based on position information common to a plurality of transposed Bloom filters specified by the first specification step;
The search method characterized by performing.

Images