JP2020119363A - Retrieval device and method for creating hash table - Google Patents

Abstract

To reduce occurrence of hash collision even when an original space of a hash method is large.SOLUTION: A retrieval device includes: an input unit for receiving an input of a data group to be registered in a table; a division unit for dividing each input data group into a plurality of sections to generate divided data with a predetermined number of bits; and a table creation unit that creates a hash table that associates a hash value of the prescribed number of bits with a NULL as a pointer at the time of registration of the data group, updates the pointer associated with the hash value of the divided data in the created hash table from NULL to a pointer to the hash table of the divided data following the divided data in data of the division source of the divided data, and reduces a total amount of used memories of each of the hash tables by constructing a tree whose node is each of the hash table to reduce hash collision in the hash table.SELECTED DRAWING: Figure 1

Description

The present invention relates to a search device and a hash table creation method.

Conventionally, in packet transfer and the like, a hash method has been used to search which rule applies to which packet. This hash method is a technology that realizes high-speed search.

ALAXALA, [January 13, 2019 search], Internet <URL: https://www.nic.ad.jp/en/materials/iw/2012/proceedings/d1/>

Here, when the original space of the hash method is large, the reduction scale of the hash space has to be increased due to a resource problem such as memory, and there is a risk of hash collision depending on the reduction scale.

For example, when the hash space used for search is Nbit, the identity function (y=x) is used as the hash function, and a 64-bit machine is used, the hash table requires 2 ^N × 64 (bit) memory resources. Is. Here, in order to suppress the memory resource of the hash table within 16 GByte, the hash space must be 31 bits or less. Therefore, when the original space of the hash method is 64 bits, a hash function with a reduction scale of 1/(2 ³³ ) must be selected. As a result, if the number of rules (data) to be registered is 1 billion or less, all the data may be mapped to the same hash value. In that case, all registered data must be linearly searched. For example, even in the technique described in Non-Patent Document 1, since the hash space is scaled down, a plurality of data are mapped to one hash value, are not fixed, and need to be searched again.

Therefore, an object of the present invention is to solve the above-mentioned problems and reduce the occurrence of hash collision even when the original space of the hash method is large.

In order to solve the above-mentioned problems, the present invention provides an input unit that receives an input of a data group to be registered in a table, and divided data of a predetermined number of bits by dividing each input data group into a plurality of sections. And a dividing unit that generates a hash table that associates the hash value of the predetermined number of bits with NULL as a pointer when registering the data group, and corresponds to the hash value of the divided data in the created hash table. By updating the pointer attached from NULL to a pointer to the hash table of the divided data following the divided data in the original data of the divided data, and constructing a tree with each of the hash tables as a node, the hash A table creating unit that reduces the total amount of memory used for each table and reduces hash collisions in the hash table.

According to the present invention, it is possible to reduce the occurrence of hash collision even when the original space of the hash method is large.

FIG. 1 is a diagram illustrating an example of a tree of a hash table constructed by the search device according to the first embodiment. FIG. 2 is a diagram illustrating a configuration example of the search device according to each embodiment. FIG. 3 is a diagram for explaining the effect of the search device according to the second embodiment. FIG. 4 is a diagram for explaining the effect of the search device according to the third embodiment. FIG. 5 is a diagram for explaining the maximum value of the used memory amount of the tree of the hash table created by the search device according to the fourth embodiment. FIG. 6 is a diagram illustrating an example of a computer that executes a program that implements the functions of the search device according to each embodiment.

Hereinafter, modes (embodiments) for carrying out the present invention will be described separately from the first embodiment to the fourth embodiment with reference to the drawings. The present invention is not limited to each embodiment.

[First Embodiment]
[Example of tree of hash table]
First, an example of building a tree of a hash table according to the first embodiment will be described with reference to FIG. Here, a case where the hash space (original space) used for data search is 64 bits will be described as an example. The search device according to the present embodiment divides a 64-bit hash space used for search into four, and constructs a tree having a hash table indicating a 16-bit hash space as a node.

When this search device receives an input of a group of data to be registered (hereinafter appropriately referred to as “rule”) in the hash table, it newly creates a hash table necessary for registration. Here, a case where an identity function (y=x) is used as the hash function used in the hash table will be described as an example.

Further, the hash table in this embodiment is a table in which hash values and pointers are associated with each other. Note that bool (false/true) is associated with the hash values of the hash tables 104, 106, 109, and 110 shown in FIG. 1, but a pointer may be associated therewith.

For example, the search device divides the data (1) to (4) shown in FIG. 1 into four sections (sections 111 to 114), and divides the divided data (divided data) into 16-bit hash tables 101 to 110. sign up. Specifically, the search device updates the pointer corresponding to the divided data (=hash value) appearing in the data of (1) to (4) in the hash table 101 from “NULL” to “NP (next pointer)”. To do. Then, among the hash values of the hash table (for example, the hash table 102) corresponding to this NP, the pointer corresponding to the divided data following the above divided data is updated from “NULL” to “NP”.

For example, the search device creates the hash table 101 and updates the pointers of the divided data “200” and “300” in the hash table 101 from “NULL” to “NP”. Next, when the search device creates the hash table 102, the pointers of the divided data “300” and “600” following the divided data “200” in the hash table 102 are updated from “NULL” to “NP”. To do. The search device registers the data (1) to (4) in the hash tables 101 to 110 by executing the above-described processing for each of the data (1) to (4). As a result, a hash table tree in which the hash tables 101 to 110 are used as nodes and the nodes are connected by NPs is constructed.

After the search device constructs the above-mentioned hash table tree, for example, when a search request for the data of (1) is received, the search of the data of (1) is performed by tracing the tree of the hash table shown in FIG. To do. As a result of the search, when the data of (1) is found, the search device returns true, and when the data of (1) cannot be found, the search device returns false.

As described above, the search device divides the hash space to be searched, for example, and constructs a tree of a hash table indicating the divided hash space. Further, the search device creates a hash table required when registering the data. As a result, even when the hash space to be searched is relatively large, the amount of memory used in the hash table can be significantly reduced. With this, the search device does not have to scale the hash value used in the hash table, or the scale can be reduced even when the hash value is scaled, so that the occurrence of hash collision can be reduced. it can. Further, for example, in the search device, the total amount of memory that can be used for each hash table (that is, the total amount of memory that can be used for rule registration) is limited to within a predetermined value (for example, 16 GBytes). Even if there is, the used memory amount can be kept within the above-mentioned predetermined value.

[Constitution]
Next, a configuration example of the search device 10 according to the first embodiment will be described with reference to FIG. The search device 10 includes an input unit 11, a control unit 12, a storage unit 13, and an output unit 14.

The input unit 11 and the output unit 14 are realized by, for example, an input/output interface included in the search device 10. The control unit 12 is realized by, for example, a program execution process performed by a CPU included in the search device 10. The storage unit 13 is realized by, for example, a main memory and a hard disk device included in the search device 10.

The input unit 11 receives input of various data. For example, the input unit 11 receives an input of a data group to be registered in the hash table. The control unit 12 controls the entire search device 10, and includes, for example, a division unit 121, a table creation unit 122, and a search unit 123. The division control unit 124 and the instruction unit 125 indicated by broken lines may or may not be equipped, which will be described later.

The dividing unit 121 generates divided data having a predetermined number of bits by dividing each input data group into a plurality of sections.

The table creation unit 122 creates a hash table that is a registration destination of the data to be registered, and registers the data in the created hash table. Specifically, the table creation unit 122 creates a hash table in which the above-mentioned hash value having a predetermined number of bits and a pointer are associated with each other. Then, the table creation unit 122 registers the split data created by the split unit 121 in the created hash table. For example, the table creation unit 122 sets a pointer associated with the hash value of the divided data in the created hash table to a pointer to a hash table of divided data that follows the divided data in the original data of the divided data ( Update to NP) above. The table creation unit 122 repeats such processing until the divided data in the last section of the data to be registered. As a result, a hash table tree in which the hash tables to be searched are connected by the pointer (NP described above) is constructed.

The search unit 123 searches the search target data using the tree of the hash table created by the table creation unit 122, and outputs the search result via the output unit 14.

The storage unit 13 stores various data necessary for the control unit 12 to operate. For example, the storage unit 13 stores the number of divisions of the hash space and the like. The number of divisions can be appropriately changed based on an instruction input from the input unit 11, for example. The storage unit 13 also stores the hash table created by the table creation unit 122.

The output unit 14 outputs the processing result of the control unit 12. For example, the output unit 14 outputs the search result by the search unit 123.

According to such a search device 10, for example, even if the 64-bit hash space is divided into four and 10,000 rules are registered, the maximum memory usage is about 15.7 GByte. In other words, in the above case, the maximum number of hash tables is 1+10000+10000+10000=30001, so the memory usage is 30001 (the number of tables) x 65536 (the number of indexes per table) x 8 Bytes (per index) It can be stored within the capacity) ≒ 15.7 GByte.

In this way, the search device 10 newly creates only the hash table relating to the rule to be registered, so that the memory area required for the hash table can be reduced. As a result, it becomes easy to use the identity function (y=x) as the hash function. Further, when the above 10,000 rules are registered without dividing the hash space as in the prior art, it is necessary to increase the reduction scale of the hash space. Therefore, a maximum of 10,000 hash collisions occur when searching for rules, which may increase search time. On the other hand, in the case of the search device 10, when the rule is searched, the NULL determination occurs (the number of divisions of the hash space-1) times, but it is not necessary to increase the reduction scale of the hash space. As a result, since the occurrence of hash collision can be reduced, it is possible to prevent the search time from increasing.

[Second Embodiment]
Next, the search device 10 according to the second embodiment will be described. The same components as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

The table creation unit 122 of the search device 10 according to the second embodiment, in the dividing unit 121, generates divided data in which each rule group to be registered is divided into sections of a predetermined number of bits, and then the section is divided into sections. The types of divided data groups belonging to are totaled. Then, when the tree of the hash table is constructed, the table creation unit 122 arranges the hash table of the divided data in the section in which the type of the divided data that has been aggregated is smaller on the root side of the tree, and the type of the aggregated data that has been aggregated is larger. The hash table of the divided data of the section is arranged at the end side of the tree. By doing so, the search device 10 can reduce the memory usage of the tree of the hash table. As a result, the search device 10 can increase the number of rules that can be registered in the tree of the hash table. For example, even if the total amount of memory that can be used for rule registration in the search device 10 is limited to within a predetermined value (for example, 16 GBytes), more rules can be registered.

For example, when the rule shown in (1) to (4) of FIG. 1 is divided into four sections (sections 111, 112, 113, 114), the section 111 has the smallest kind of divided data (2 kinds. ) Sections 112 and 114 have the second smallest kind of divided data (3 kinds), and section 113 has the largest kind of divided data (4 kinds).

Therefore, the table creation unit 122 arranges the hash table of the divided data of the section 111 on the root side of the tree, and the hash table of the divided data of the section 113 on the end side of the tree. A hash table of 114 divided data is arranged between the root and the end of the tree.

In addition, the search device 10 calculates an example of the amount of memory used when registering 100,000 rules in which the number of types of divided data for each section when 64-bit data is divided into four is 1, 10, 1000, and 50000, respectively. This will be described with reference to FIG.

For example, the hash table of the divided data group in the section where the number of types of divided data is “50000” is arranged on the root side of the tree, and the number of types is “1000” in ascending order of the number of types toward the end of the tree. Consider a case where a hash table of divided data in a section→a hash table of divided data in a section of which the number of types is “10”→a hash table of divided data in a section of which the number of types is “1” are arranged in this order. In this case, the maximum number of tables in each section is 1, 50000, 100000, 100000, as shown in FIG. 3A, so the total number of hash tables is 1+50000+100000+100000=250001 at maximum. is there. Therefore, the maximum memory usage of the tree of the hash table is about 131.1 GByte, which may not be within 16 GByte.

On the other hand, the table creation unit 122 of the search device 10 arranges the hash table of the divided data in the section where the number of types of the divided data is “1” on the root side of the tree, and the ascending order of the number of types toward the end of the tree. In this order, the hash table of the divided data in the section where the number of types is "10" → the hash table of the divided data in the section where the number of types is "1000" → the hash table of the divided data in the section where the number of types is "50000" are arranged in this order. .. In this case, the maximum number of tables in each section is 1, 1, 10, and 10000 as shown in FIG. 3B, so the total number of hash tables is 1+1+10+10000=10011 at maximum. is there. Therefore, the maximum memory usage of the tree of the hash table is about 5.2 GByte, which is within 16 GByte.

[Third Embodiment]
Next, the search device 10 according to the third embodiment will be described. The same configurations as those of the above-described embodiment are denoted by the same reference numerals, and description thereof will be omitted.

As described in the first embodiment, as the number of divisions of the hash space (for example, four divisions in the first embodiment) is increased, the memory area used by the hash table tree can be saved. Therefore, the maximum memory area (threshold value) used by the tree of the hash table is specified in the search device 10 in advance by the config or the like, and the total amount of memory area (used memory amount) used by the tree of the hash table is set to the above threshold value. When it exceeds, the search device 10 may increase the number of divisions.

For example, in the third embodiment, the search device 10 further includes a division control unit 124 shown by a broken line. The division control unit 124 increases the number of divisions of the hash space above a predetermined number when the total amount of used memory of the tree of the hash table exceeds a predetermined threshold.

For example, when the partition control unit 124 determines that the total amount of used memory of the tree of the hash table exceeds the predetermined threshold value, the number of partitions used by the partition unit 121 and the table creation unit 122 was initially “4”, Increase to "8". For example, the division control unit 124 instructs the division unit 121 to generate division data by dividing each rule to be registered into 8 bits, and associates the table creation unit 122 with the 8-bit hash value and the pointer. Instruct to create a hash table.

With this, the search device 10 can keep the total amount of used memory of the tree of the hash table within a predetermined threshold even if the number of rules to be registered is large. As a result, for example, when the total amount of memory that can be used for rule registration in the search device 10 is limited to within a predetermined threshold value (for example, 16 GByte), even when the number of rules to be registered increases, This can be dealt with.

For example, consider a case in which the search device 10 registers 100,000 rules in the tree of the hash table when the total amount of memory that can be used for rule registration is limited to within 16 GBytes. In this case, if the search device 10 sets the number of divisions of the hash space to “4”, for example, the number of hash tables is 1+65536+100000+100000=265537 at maximum, as shown in FIG. is there. Therefore, the maximum memory used in the tree of the hash table is about 139.2 GBytes, which does not fit within 16 GBytes. As a result, the search device 10 may not be able to register the rule to be registered.

On the other hand, if the number of divisions of the hash space is set to “8” (that is, divided into 8 bits each) as in the search device 10 of the third embodiment, for example, as shown in FIG. The maximum number of tables is 1+256+65536+100000+...+100000=565793. Therefore, the memory usage of the tree of the hash table is about 1.2 GByte at maximum, and can be kept within 16 GByte. As a result, the search device 10 can register the rule to be registered.

[Fourth Embodiment]
Next, the search device 10 according to the fourth embodiment will be described. The same configurations as those of the above-described embodiment are denoted by the same reference numerals, and description thereof will be omitted.

As described above, the search device 10 can also save the memory area used by the tree of the hash table by increasing the reduction scale of the hash function. Therefore, when the maximum memory area (threshold value) used by the tree of the hash table is specified in the search device 10 in advance by the configuration and the total amount of used memory of the tree of the hash table exceeds the above threshold value, the search device 10 The reduction scale of the hash function may be increased.

Here, to increase the reduction scale of the hash function means, for example, when the search device 10 uses mod N (N is a natural number) for the hash function, the value of N is made smaller than the value used so far. Alternatively, if the search device 10 originally uses the identity function for the hash function, mod N (N is a natural number) is used for the hash function. An embodiment in this case will be described as a fourth embodiment.

For example, as described in the section of the third embodiment, when the search device 10 registers 100,000 rules and the identity function is used as the hash function, the total amount of used memory of the tree of the hash table exceeds 16 GByte. there is a possibility. However, even if the search device 10 registers 100,000 rules by using y=x mod 8192 (scale 1/8) as the hash function, the maximum memory usage of the tree of the hash table is about With 13.6GByte, it can be suppressed to. This will be described with reference to FIG.

For example, as shown in FIG. 5, when the search device 10 divides a 64-bit hash interval into four and registers 100,000 rules, mod 8192 (scale 1/8) is applied as a hash function used for each hash table. At this time, the maximum number of hash tables is 1+8192+100000+100000=208193. Therefore, the memory usage of the tree of the hash table is about 13.6 GByte at maximum, which is within 16 GByte.

[Other Embodiments]
Note that the search device 10 increases the number of divisions of the hash space when the total amount of used memory of the tree of the hash table exceeds a predetermined threshold value (see the third embodiment), or the hash function used for the hash table. The user of the search device 10 may set whether to increase the reduction scale (see the fourth embodiment).

In this case, when the total amount of used memory of the tree of the hash table exceeds a predetermined threshold, the search device 10 inputs the setting information indicating whether to increase the number of divisions of the hash space or the reduction scale of the hash function. When it is accepted via 11, the setting information is stored in the storage unit 13. Then, when the total amount of used memory of the tree of the hash table exceeds a predetermined value, the instruction unit 125 (see FIG. 1) of the search device 10 causes the division control unit 124 to divide the number of divisions according to the setting information stored in the storage unit 13. Is increased above a predetermined value, or the table creation unit 122 is instructed to increase the reduction scale of the hash function used in the hash table above a predetermined value.

By doing so, when the total amount of used memory in the tree of the hash table exceeds a predetermined value, the search device 10 adopts a method of increasing the number of divisions of the hash space (see the third embodiment), or The user of the search device 10 can specify whether to adopt the method of increasing the reduction scale of the hash function used in the table (see the fourth embodiment).

The search device 10 may determine the number of divisions of the hash space and the reduction scale of the hash function used for each hash table according to the total amount of used memory of the tree of the hash table. For example, the search device 10 increases the number of divisions of the hash space as the total amount of used memory of the tree of the hash table increases (that is, the number of registered rules increases), or the number of hash functions used for each hash table increases. The reduction scale may be increased.

[program]
Further, it can be implemented by installing a program that realizes the function of the search device 10 described in the above embodiment into a desired information processing device (computer). For example, the information processing device can be caused to function as the search device 10 by causing the information processing device to execute the above-described program provided as package software or online software. The information processing device mentioned here includes a desktop or notebook personal computer, a rack-mounted server computer, and the like. Further, in addition to the above, the information processing apparatus includes a mobile communication terminal such as a smartphone, a mobile phone, a PHS (Personal Handyphone System), and a PDA (Personal Digital Assistant) in its category. Further, the search device 10 may be mounted on a cloud server.

An example of a computer that executes the above program will be described with reference to FIG. As shown in FIG. 6, the computer 1000 has, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These units are connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM (Random Access Memory) 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. A mouse 1110 and a keyboard 1120 are connected to the serial port interface 1050, for example. A display 1130, for example, is connected to the video adapter 1060.

Here, as shown in FIG. 6, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. The various data and information described in the above embodiments are stored in, for example, the hard disk drive 1090 or the memory 1010.

Then, the CPU 1020 reads the program module 1093 and the program data 1094 stored in the hard disk drive 1090 into the RAM 1012 as necessary, and executes the above-described procedures.

The program module 1093 and the program data 1094 related to the above program are not limited to being stored in the hard disk drive 1090, and may be stored in a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. May be issued. Alternatively, the program module 1093 and the program data 1094 related to the above program are stored in another computer connected via a network such as LAN or WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070. May be done.

10 search device 11 input unit 12 control unit 13 storage unit 121 division unit 122 table creation unit 123 search unit 124 division control unit 125 instruction unit

Claims (7)

An input unit that receives the input of the data group to be registered in the table,
A division unit that generates division data of a predetermined number of bits by dividing each input data group into a plurality of sections,
When registering the data group, create a hash table that associates the hash value of the predetermined bit number with NULL as a pointer, and from the NULL pointer that is associated with the hash value of the divided data in the created hash table, The total amount of used memory of each of the hash tables is constructed by updating the pointers to the hash tables of the divided data following the divided data in the original data of the divided data and constructing a tree with each of the hash tables as a node. And a table creation unit that reduces hash collisions in the hash table,
A search device comprising: The table creation unit,
For each section, the types of the divided data belonging to the section are aggregated, and the hash table of the divided data of the section having the smaller types of the aggregated divided data is arranged on the root side of the tree, and the aggregated divided data is arranged. 2. The search according to claim 1, wherein the number of data that can be registered is increased by constructing the tree arranged on the end side of the tree for a hash table of divided data in a section having a large number of types. apparatus. The table creation unit,
The search apparatus according to claim 1, wherein a hash collision is not generated in the hash table by using an identity function as a hash function used in the hash table. The search device according to claim 1, further comprising a division control unit that increases the number of the divided sections above a predetermined value when the total amount of used memory of the tree exceeds a predetermined value. The table creation unit,
The search device according to claim 1, wherein when the total amount of memory used by the tree exceeds a predetermined value, the reduction scale of the hash function used in the hash table is increased above a predetermined value. When the total amount of memory used by the tree exceeds a predetermined value, an instruction to the division control unit to increase the number of the division sections above a predetermined value according to the setting information stored in the storage unit, or the table creation unit The search device according to claim 4, further comprising an instruction unit that issues an instruction to increase a reduction scale of a hash function used in the hash table above a predetermined value. A method for creating a hash table executed by a search device, comprising:
A step of receiving the input of the data group to be registered in the table,
Generating divided data with a predetermined number of bits by dividing each input data group into a plurality of sections,
When registering the data group, create a hash table that associates the hash value of the predetermined bit number with NULL as a pointer, and from the NULL pointer that is associated with the hash value of the divided data in the created hash table, The total amount of used memory of each of the hash tables is constructed by updating the pointers to the hash tables of the divided data following the divided data in the original data of the divided data and constructing a tree with each of the hash tables as a node. To reduce hash collisions in the hash table,
A method of creating a hash table characterized by including.

Images