KR20210052148A - Method and apparatus for conversing data key value - Google Patents

Abstract

The present invention relates to a method for performing dynamic hashing scannable by a processor. The method may comprise the steps of: determining a distribution form of key value distribution information calculated for data key values; determining a transformation function which transforms the data key values based on a distribution shape of the key value distribution information; converting the data key values into values corresponding to a uniform distribution using the determined conversion function; and generating a hash index structure related to the converted data key values. The present invention realizes fast performance in all insert, search, and scan functions by converting an original data key value based on a conversion function.

Description

Data key value conversion method and device {METHOD AND APPARATUS FOR CONVERSING DATA KEY VALUE}

Hereinafter, a method and an apparatus for converting a data key value are provided.

Traditional storage systems, such as LSM-based key value storage, are designed for slow disks. In the case of a system using a hard disk drive or an SSD, disk I/O latency causes a bottleneck, and the index structure of such a system can provide minimal performance. For example, the index query time for a B +- tree index may be less than 1% of the data block read time. Recently, nonvolatile memory that can address bytes such as STT-MRAM, fast storage systems such as the currently commercially available Optane DC Persistent Memory, and ultra-low latency SSDs and Optane SSDs such as Z-SSD are changing the environment. In the case of such a fast storage system, the performance of the software layer, such as the efficiency of the index structure, becomes important, and accordingly, the final latency of the system can be affected more greatly.

In the case of storage systems, two widely used indexing structures are B +-tree and hash index. B+-tree is a dynamic structure that can adjust its internal structure based on the current workload, and it can support not only common tasks such as inserts and searches, but also scan tasks. On the other hand, the hash index can generally provide faster insertion and search operations compared to the B +-tree. However, hash indexes cannot provide scan operations.

While providing scan operations, there have been attempts to support insertions and searches as fast as hash indexes. For example, a hybrid index structure using both indexes has been proposed. However, there was a problem that there was an overhead in maintaining the two index structures, including the problem of consistency between the two index structures.

Korean Patent Publication No. 10-2006-0013815 (Publication date: February 14, 2006)

According to an embodiment, a method of performing scannable dynamic hashing by a processor includes: determining a distribution type of key value distribution information calculated for data key values; Determining a conversion function for converting the data key values based on a result of determining the distribution type of the key value distribution information; Converting the data key values into values corresponding to a uniform distribution using the determined transformation function; And generating a hash index structure related to the converted data key values.

The method of performing dynamic hashing may further include determining whether to convert the data key values based on a result of calculating a distance between a value related to a uniform distribution and the key value distribution information.

Determining whether to convert the data key values may include calculating a difference between a variance value of the uniform distribution and a variance value of the key distribution information as the distance.

The determining whether to convert the data key values may include comparing the calculated distance to a predetermined threshold distance; And converting the data key values based on a result of comparing the calculated distance and the threshold distance.

The method of performing dynamic hashing may further include determining whether to convert the data key values based on a cluster level of the data key values.

The determining whether to convert the data key values may include determining the number of common bits between the data key values as the cluster level; And converting the data key values in response to the cluster level being equal to or greater than a predetermined threshold level.

The key value distribution information may include a variance and an average of the data key values.

The determining of the distribution shape may include determining whether the distribution shape and a shape corresponding to each continuous probability distribution of a plurality of continuous probability distributions match; And in response to a case in which the distribution shape matches a shape corresponding to one of the plurality of continuous probability distributions, determining the distribution shape as the matched continuous probability distribution.

The determining of the distribution type may include determining a cluster level of the data key values in response to a case where the distribution type does not match the plurality of continuous probability distributions; And in response to the determined cluster level being less than a predetermined threshold level, skipping conversion of the data key value.

The determining of the transformation function may include determining the transformation function based on a cumulative distribution function (CDF) for a distribution type of the key value distribution information.

The determining of the transformation function may include transforming the cumulative distribution function into a set of linear functions based on a linear interpolation method.

The determining of the conversion function may include dividing the data key values into a plurality of groups according to the number of data key values, and calculating a linear function for each group.

The determining of the hash index structure includes creating a directory related to the data key value and a bucket corresponding to the directory, and mapping the converted data key values to the bucket according to the conversion function. It may include steps.

The apparatus for performing scannable dynamic hashing according to an embodiment determines a distribution type of key value distribution information calculated for data key values, and based on a result of determining the distribution type of the key value distribution information, the data key value A hash index structure related to the transformed data key values is determined by determining a transform function that transforms the data keys, transforming the data key values into values corresponding to a uniform distribution using a predetermined transform function. A processor that generates ); And a memory for storing the converted data key values in the hash index structure.

According to an embodiment, the data key value conversion method converts the original data key value based on a conversion function, thereby providing fast performance in all of the insert, search, and scan functions.

In conventional hashing, the hashing functions distribute the keys evenly and map them so that collision does not occur. Therefore, the converted data key and the original data key are not sorted in the same order. On the contrary, according to an embodiment, the processor may store the converted converted data keys in the same order as the original data keys and sorted in the bucket.

1 is a flowchart illustrating a method of determining an index structure by converting a key value according to an embodiment.
2 is a diagram illustrating a hashing method of extended hashing according to an embodiment.
3 is a diagram illustrating an index structure in which a key value is converted based on distribution information of a key value according to an embodiment.
4 is a flowchart illustrating a method of determining whether to convert a key value according to an embodiment.
5 is a diagram illustrating a distribution of key values according to an embodiment.
6 is a diagram illustrating creating a directory according to an embodiment.
7 is a block diagram showing an outline configuration of an apparatus for converting a key value according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for the purpose of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to a specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea.

Although terms such as first or second may be used to describe various components, these terms should be interpreted only for the purpose of distinguishing one component from other components. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

When a component is referred to as being "connected" to another component, it is to be understood that it may be directly connected or connected to the other component, but other components may exist in the middle.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, action, component, part, or combination thereof is present, but one or more other features or numbers, It is to be understood that the presence or addition of steps, actions, components, parts, or combinations thereof, does not preclude the possibility of preliminary exclusion.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the relevant technical field. Terms as defined in a commonly used dictionary should be construed as having a meaning consistent with the meaning of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in the present specification. Does not. Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The same reference numerals shown in each drawing indicate the same members.

1 is a flowchart illustrating a method of determining an index structure by converting a key value according to an embodiment.

The processor according to an embodiment may perform a method of performing scannable dynamic hashing. In step 110, the processor may determine a distribution type of the key value distribution information calculated for the data key values. The processor may cluster each of the key values in units of a predetermined number of bits among a plurality of bits listed as "1" and "0", and determine a key value distribution based on the number of data included in the cluster. For example, the key value distribution may be a distribution determined based on clusters clustered in units of the highest predetermined number of bits.

In operation 120, the processor may determine a transformation function for changing data key values based on a result of determining the distribution type of the key value distribution information. For example, when the key value distribution information matches the continuous probability distribution, the processor may determine a transformation function based on a cumulative distribution function for a distribution type of the key value distribution information. In this case, the processor can transform the cumulative distribution function into a set of linear functions based on the linear interpolation method. By determining the transformation function based on the set of linear functions, the processor is faster than when the linear function is not used. You can calculate the transformation function. The processor divides it into a plurality of groups to calculate a linear function, and may divide it into a plurality of groups according to the number of data key values. For another example, when the key value distribution information does not match the continuous probability distribution, the processor may determine a transform function as a function that increases the slope of the cumulative distribution function in a section in which the clustered data key values exist.

In step 130, the processor may convert the data key values into values corresponding to a uniform distribution by using the determined transformation function.

In step 140, the processor may generate a hash index structure related to the converted data key values. The processor may create a directory related to the data key value and a bucket corresponding to the directory, and map the converted data key values to the bucket according to the conversion function. In this case, the mapped key values may be arranged in the same order as the key values before conversion according to the conversion function. The generation of the hash index structure will be described in detail later with reference to FIG. 6.

According to an embodiment, the processor may determine whether to convert data key values before determining a distribution type of the key value distribution information in order to determine a conversion function. Determining whether to convert data key values will be described later in FIG. 4.

2 is a diagram illustrating a hashing method of extended hashing.

Extended hashing is a dynamic indexing structure that can increase or decrease when the workload changes dynamically, and may differ from the traditional hashing structure that requires rehashing. In the embodiment shown in FIG. 1, n is the number of bits used in the key, and a hash value of a given key K may be used as a pseudo-key. Here, as K` = h(K), h may be a hash function.

In extended hashing, a hash table can include a directory and a bucket. The directory may be an array in which a prefix related to the most significant bit (MSB) or a suffix related to the least significant bit (LSB) of a pseudo key is used as an index. According to FIG. 2, the processor may use the most significant two bits of a pseudo key as an index of a directory. The processor may set a global depth G corresponding to the number of most significant bits used as an index in the directory. That is, G (where G is a natural number) can determine the size of the directory.

According to FIG. 2, each item in a directory may mean a bucket capable of storing a fixed number of key values. The key value may be a pointer to an actual value associated with the key. When the processor inserts the key value, it uses the most significant bit of the hash to find the directory entry, and if it finds the directory entry, and if there is free space in the bucket pointed to by the directory entry, it stores the key value and key value in the free space of the bucket, and Related data can be saved. For example, FIGS. 110 and 120 may show states before and after inserting the "01101 ..." key into the hashing index, respectively. The processor may set a local depth L in each bucket, where the local depth may be greater than or equal to 0 and less than or equal to the global depth G. Any key starting with the L most significant bits of the directory index linked to this bucket can be included. Accordingly, according to FIG. 120, two or more items in a directory may indicate the same bucket.

If the bucket is overfilled, the processor can split the bucket into two buckets. There may be two situations in which the buckets are separated. The first is when L <G, and this may include a case in which several directory entries point to the rightmost bucket as in the case of FIG. 120. In this case, the processor can keep G the same, increment L in both buckets by one of the original buckets, and adjust the contents of the original bucket according to the value of L + 1 most significant bit of the pseudo key. For example, the bucket pointed to by dir[2] and dir[3] in FIG. 120 is divided into two buckets in FIG. 130, so that a key with a prefix of "10" is stored in one bucket, and a prefix of "11" is another. Can be stored in a bucket.

The second situation may be the case where G = L. In this case, one directory entry points to one bucket. Therefore, the directory must be expanded to accommodate partitioning, which is called directory doubling. To this end, the processor may first increase the number of bits used as an index, that is, G, and divide the content and adjust the L value in the same manner as in the first case. For example, in FIG. 130, when one or more keys are added to the bucket indicated by dir[1], the processor multiplies the directory, so that the L value of the bucket becomes 3, and dir[1] becomes dir[2] and dir[3. Can be divided into ].

3 is a diagram illustrating an index structure in which a key value is converted based on distribution information of a key value according to an embodiment.

According to one embodiment, the processor is based on extended hashing, but by using a remapped key rather than a hash key as a pseudo key, a scannable key value is converted while maintaining the original order of the keys. It can provide a dynamic hashing method. 3 shows the architecture of a key value conversion method. For example, an indexing structure using a directory and a bucket maintained in a persistent memory (PM) may be extended hashing using a bucket that is basically aligned. However, the memory in which the indexing structure is stored is not limited to nonvolatile memory, but may also include volatile memory. The nonvolatile memory may include, for example, a hard disk and a flash memory, as well as PM and MRAM. Volatile memory may include DRAM and SRAM.

Instead of using a hash function to generate a virtual key, the processor can instead use the raw key itself as a pseudo key and then store the keys in each bucket in sorted order. This enables efficient scanning. However, if a large number of keys exist in a specific range and only a few keys exist in the directories in the rest of the range, such a non-uniform key distribution can lead to an unbalanced directory, and directory doubling is performed unnecessarily. You can do it.

For example, the processor may evenly redistribute keys that follow a certain distribution while maintaining the original order of the data keys. In particular, the processor may transform the key value so that the key value distribution becomes uniform based on a probability integral transform. The processor may transform the random variable so that the continuous distribution of the given random variable follows a standard uniform distribution based on the probability cumulative transformation. At this time, the standard uniform distribution has a range (e.g., 0 to 1), but it ^{is expanded to the entire key range (e.g., 0 to 2 n} -1) to follow a uniform distribution. Can be converted. That is, when the random variable X has a continuous distribution _{, the random variable Y = F X (X), where F X} is a cumulative distribution function (CDF) for the distribution, may follow a uniform distribution.

For another example, in response to a case in which the key value distribution of the data key values does not match the plurality of continuous probability distributions, the processor may calculate the clustering level of the data key values. The processor may determine that the data key values are clustered in response to the clustering level being equal to or greater than the threshold level. The processor may increase the slope of the cumulative distribution function of a section in which the clustered data key values exist. Since the key distance in the corresponding section is increased by increasing the slope of the cumulative distribution function, the processor can uniformly redistribute the key value distribution.

According to an exemplary embodiment, a processor that converts a key value based on a conversion function and generates a hash index structure includes a key distance checker, a cumulative distribution function tuner, and a key rebuild according to the function. It can be classified as a key remapper. The units may be executed by the same processor, and may be divided according to functions.

Determining whether to convert a key value of data by calculating at least one of a uniform distribution, a distance between key distributions, and a cluster level by the inter-key distance checker will be described later in detail with reference to FIG. Determining the distribution shape of the key value by the cumulative distribution function tuning unit will be described later with reference to FIG. 5. Converting a data key value by the key remapping unit and generating a hash index structure will be described later with reference to FIG. 6.

4 is a flowchart illustrating a method of determining whether to convert a key value according to an embodiment.

When directory doubling occurs more than a threshold number of times, the processor may determine how far the data key value distribution is from the uniform key value distribution. According to an embodiment, the processor may determine whether to perform dynamic hashing, for example, converting data key values, based on a distance between data key values and a cluster level. However, the trigger condition for the operation of determining whether the conversion operation is performed is not limited to the number of directory doublings. For another example, the processor may periodically determine how far the distribution of the data key values is apart from the uniform distribution of the key values.

According to an embodiment, the processor may determine whether to convert data key values based on a result of calculating a distance between a value related to a uniform distribution and the key value distribution information. According to another embodiment, the processor may determine whether to convert data key values based on a cluster level of data key values. In FIG. 4, the processor selectively calculates one of a uniform distribution and a distance between the key value distribution information and a cluster level of data key values according to whether the key value distribution information and the continuous probability distribution match, and the calculated distance or the calculated cluster An example of determining whether to perform conversion of data key values using a level will be described. However, this is purely exemplary and is not limited thereto, and the processor may determine whether to start the conversion operation by performing only steps 412 and 413, or determine whether to start the conversion operation by performing only steps 414 and 415. have.

First, in step 411, the processor may determine whether the distribution type of the key value distribution information matches the continuous probability distribution. For example, the processor may determine whether a distribution type of the key value distribution information and a type corresponding to each continuous probability distribution of a plurality of continuous probability distributions are matched. The processor may calculate a similarity between a distribution type of the key value distribution information and a type corresponding to each continuous probability distribution. The processor may determine that the distribution shape matches the shape corresponding to the continuous probability distribution, for example, in response to a case in which the calculated similarity for an arbitrary continuous probability distribution exceeds a threshold similarity. In response to a case where the distribution shape matches a shape corresponding to one of the plurality of continuous probability distributions, the processor may determine the distribution shape as the matched continuous probability distribution.

In step 412, in response to the case where the distribution type matches the continuous probability distribution, the processor may determine a distance between the uniform distribution and the key value distribution information. For example, the processor may calculate a difference between a variance value of a uniform distribution as a distance and a variance value of the key distribution information. The key distribution information may be distribution information of an original key value.

Subsequently, in step 413, the processor may compare the calculated distance to a predetermined threshold distance. The processor may convert data key values based on a result of comparing the calculated distance and the threshold distance. For example, the processor may determine whether the determined distance is equal to or greater than a predetermined threshold distance. If the determined distance is greater than or equal to a predetermined threshold distance, the processor may proceed with a process for converting the data key value, but if it is less than the threshold distance, the processor may determine that the key value distribution is uniform and not perform the key value conversion.

For example, the processor may use the variance of the key value distribution to determine how far the key value distribution deviates from the uniform distribution. If the variance of the key value distribution is greater than the critical variance, it may mean that the data key value is wider from the average, and if the variance of the key value distribution is smaller than the critical variance, it may mean that the key value is close to the mean. have.

Also, the processor may determine whether to convert data key values based on a cluster level of data key values. For example, in step 414, the processor may determine the clustering level of the data key values in response to the case where the distribution shape does not match the plurality of continuous probability distributions. The processor may determine the number of common bits between data key values as the cluster level. The number of common bits may indicate the number of bit positions in which the same bit value appears. For example, the number of common bits may indicate the number of bit positions at which a common bit value appears from the most significant bit position. In other words, the number of common bits in the data key values may represent the number of bit positions from the most significant bit position to the bit position immediately before the next lower bit position where different bit values appear in the corresponding data key values. For example, when the data key values are "1010" and "1011", since the bit sequence of "101" from the most significant bit is common, the number of common bits may be 3. For another example, when the data key values are "1000" and "1010", since the bit sequence of "10" from the most significant bit is common, the number of common bits may be 2.

And in step 415, the processor may compare the cluster level to the threshold level. For example, in response to the cluster level being equal to or greater than a predetermined threshold level, the processor may proceed to step 120 to continuously perform conversion of data key values. When the cluster level is the number of common bits, the threshold level may represent a predetermined threshold common number. For another example, in response to a case where the determined cluster level is less than a predetermined threshold level, the processor converts the data key value (e.g., determines the conversion function in step 120 of FIG. 1, the key value of step 120). Transformation and generation of the hash index structure in step 140) may be skipped.

5 is a diagram illustrating a distribution of key values according to an embodiment.

The first graph 510 shows a normal distribution for the mean and variance values, and the second graph 520 shows a cumulative distribution function corresponding to the normal distribution. The third graph 530 is a graph showing the gamma distribution according to the shape k.

If the processor determines that the key value distribution is not uniform, it may convert the data key value so that the key value distribution becomes uniform. In order to convert the key value, the processor may need to determine the distribution of the data key value or the distance between the keys. The processor may determine a distribution type of the key value distribution information as one of a plurality of continuous probability distributions.

The continuous probability distribution may include, for example, a normal distribution, an exponential distribution, and a gamma distribution. In the present specification, a normal distribution, an exponential distribution, and a gamma distribution are mainly exemplarily described as a continuous probability distribution, but these are for convenience of description and are not limited thereto. For example, each of the exemplary distributions described above may have many relationships with other continuous distributions, and may be various distributions according to specific parameters. As an example, the normal distribution is a normal continuous probability distribution, and the gamma distribution with two parameters, a shape parameter k and a scale parameter q, is an exponential distribution, an Erlang distribution, and a chi-square distribution ( chi-squared distribution). Therefore, the processor can provide various workloads in consideration of various distribution types.

For example, in the case of classifying one of the continuous probability distributions into one of normal, exponential, and gamma distributions, the processor determines the distribution type of the key value distribution information based on the principle of maximum entropy. You can determine whether they are similar or not. For example, if the variance of the key value distribution information is approximated to a squared value of the average value, the processor may determine that the distribution form of the key value distribution information is an exponential distribution.

In addition, if a negative value is included when a significant range in which the key can exist in the key value distribution support set determined to be a normal distribution, it can be determined as a gamma distribution. For example, when the average and standard deviation of the normal distribution are used to find the value of the interval in which the key corresponding to 99.7% of the total exists, if the interval contains a negative range, consider that the total key distribution has only positive values. In order to reduce the error, it can be determined as a gamma distribution.

The processor classifies the distribution type of the key value distribution information into one of a plurality of continuous probability distributions (e.g., normal distribution, exponential distribution, and gamma distribution), and calculates the distribution parameter to distribute the distribution. The cumulative distribution function of can be calculated.

Additionally, the processor can transform the cumulative distribution function into a set of a series of linear functions based on linear interpolation. We can divide the subranges so that the entire range of keys from 0 to 2 ^{n -1 contains the same number of keys in each subrange.} This subrange can be found using the Quantile function of the distribution (i.e. the inverse cumulative distribution function). In the case of a continuous probability distribution that does not have a simple closed form for the quantile function, the processor can determine the transform function using an approximation based on a numerical method. The processor may generate a linear function for each subrange based on the cumulative distribution function of the distribution and use the linear function as a transformation function for the keys of the subrange.

6 is a diagram illustrating creating a directory according to an embodiment.

According to an embodiment, after the processor determines the distribution type of the key value distribution information and generates a transformation function based on the cumulative distribution function, the transformed key must be remapped. The processor can create a new directory and map the converted key value to the bucket based on the directory.

The processor may create a new directory based on the conversion function and determine the local depth of the bucket pointed to by each entry in the new directory. The processor can create a binary buddy tree that is associated with the history of the bucket for the directory for extended hashing to the most significant bit. By taking advantage of the fact that the number of consecutive directory entries pointed to by the bucket in the binary buddy tree ^{must be 2 GL while the} processor creates a new directory, it can determine the local depth of the bucket.

In FIG. 6, an array indicating a previous directory is illustrated as dir, and an array indicating a new directory is illustrated as dir _new . Each item in the array pointing to a directory can correspond to an arbitrary bucket and can be represented as a directory item. For example, dir[i] is the i-th directory item of the previous directory, and dir _new [j] is the new directory item. Can represent the j-th directory entry in the directory. Each of i and j may be an integer greater than or equal to 0.

In the directory mapping relationship 610 according to an embodiment, the processor _{may create a new directory (dir new} ) of G=4 based on the conversion function by constructing a partitioning history capable of distributing an _{existing bucket to dir new.} . Bucket B1 that dir [0] in the previous directory indicated in this embodiment, it is assumed to be indicated by the six items to _new dir [5] from the _new dir [0] for the new directory. An exemplary mapping process for the directory mapping relationship 610 will be described below. In this case, the _{minimum possible L of dir new} [0] is 0, and the current maximum 2 ⁴ = 16 directory entries can point to B1 of _{dir new [0].}

Thereafter, as in the second mapping process 630, the processor may calculate an entry in the new directory pointing to the next directory B2 (eg, the second bucket) in dir. First, based on the previous transformation function, the processor may calculate the minimum key k that can be inserted into B2. The processor can then use a new conversion function to convert k to k'and compute the index of _{dir new by extracting the prefix of k'.} dir _new case of [6] instructs the bucket B2, B1 is L = 2 levels of because have a brother bucket of B3 B1 is in the dir _new [0] dir _new [3] to be displayed in four consecutive entries have. Since B1 is divided twice, L for 4 items can be calculated as 2. Accordingly, as shown in the second mapping process 630, B3, which is a sibling bucket of B1, may exist at the level of L=2, and the processor is dir _new [4], which is the leftmost item in the subtree rooted in B3. You can map B3 to Also, at the level where L = 1, there must be a sibling bucket of B1, B4, and the processor _{can map B4 to dir new} [8] in a similar manner.

As in the third mapping process 640, since B3 needs to be divided once more, two items dir _new [4] and dir _new [5] indicate B3, and L of B3 becomes 3.

While creating a new directory, the processor can perform all operations normally except inserts and deletes.

In the case of PM, when dir _new is written, the processor can continuously call instructions (eg, CLFLUSH and MFENCE instructions) that ensure the order in which data is stored.

Since a plurality of keys mapped to the same bucket in the previous directory may have different prefixes according to the new conversion function, the processor may convert the plurality of key values and map them to different buckets in the new directory. In order to handle the case were more keys are mapped to the same bucket processor generates a new bucket for dir _new and key in any bucket in the dir - may assign a pointer to the record (key-pointer record) to the new directory. For each bucket in the directory, the processor reads the value of each key along with the bucket's associated pointer, and can map each key to a new directory one by one. The processor can use a conversion function to convert k to k and compute the prefix of k to find the index i in _{dir new.} If the bucket _{is not assigned to dir new} [i], the processor can create a new bucket and store it in memory. The processor can then update the pointer to the bucket of _{dir new} [i] using the CLFLUSH and MFENCE instructions, for example in the case of PM. If the processor moves all the key pointer records in dir to dir _new , the processor can switch to the new directory instead of the old one and use it. In the case of PM after switching to the new directory, the old directory array and the old bucket are obsolete objects. Thus, in the end, garbage may be collected by a permanent memory manager such as PMDK.

While the processor is migrating records from the old directory to the new directory, the processor can still distinguish between the key pointer records copied to the new directory and the key pointer records that were before the copy, so the hashing index is still accessed using both the old and new directories. You can search and scan.

7 is a block diagram showing an outline configuration of a key value conversion apparatus 700 according to an embodiment.

The key value conversion apparatus 700 according to an embodiment may include a processor 710 and a memory 720. The processor 710 determines a distribution type of the key value distribution information calculated for the data key values, determines a transformation function for converting data key values based on the distribution type of the key value distribution information, and a predetermined transformation function By using, data key values may be converted into values corresponding to a uniform distribution, and a hash index structure related to the transformed data key values may be generated. The memory 720 may at least temporarily store the converted data key values in the hash index structure.

The embodiments described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices, methods, and components described in the embodiments are, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to operate as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or, to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

As described above, although the embodiments have been described by the limited drawings, a person of ordinary skill in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Claims (15)

In the method of performing dynamic hashing (hashing) scannable by a processor,
Determining a distribution type of the key value distribution information calculated for the data key values;
Determining a conversion function for converting the data key values based on a result of determining the distribution type of the key value distribution information;
Converting the data key values into values corresponding to a uniform distribution using the determined transformation function; And
Generating a hash index structure related to the converted data key values
A method of performing scannable dynamic hashing comprising a. The method of claim 1,
Determining whether to convert the data key values based on a result of calculating a distance between a value related to a uniform distribution and the key value distribution information
A method of performing scannable dynamic hashing further comprising a. The method of claim 2,
The step of determining whether to convert the data key values,
Calculating a difference between the variance value of the uniform distribution and the variance value of the key distribution information as the distance
A method of performing scannable dynamic hashing comprising a. The method of claim 2,
The step of determining whether to convert the data key values,
Comparing the calculated distance to a predetermined threshold distance; And
Converting the data key values based on a result of comparing the calculated distance and the threshold distance
A method of performing scannable dynamic hashing further comprising a. The method of claim 1,
Determining whether to convert the data key values based on a cluster level of the data key values
A method of performing scannable dynamic hashing further comprising a. The method of claim 5,
The step of determining whether to convert the data key values,
Determining the number of common bits between the data key values as the cluster level; And
In response to the cluster level being equal to or greater than a predetermined threshold level, converting the data key values
A method of performing scannable dynamic hashing comprising a. The method of claim 1,
The key value distribution information includes a variance and an average of the data key values.
How to perform scannable dynamic hashing. The method of claim 1,
The step of determining the distribution shape,
Determining whether the distribution shape and the shape corresponding to each continuous probability distribution of the plurality of continuous probability distributions match; And
In response to a case in which the distribution shape matches a shape corresponding to one of the plurality of continuous probability distributions, determining the distribution shape as a matched continuous probability distribution
A method of performing scannable dynamic hashing comprising a. The method of claim 8,
The step of determining the distribution shape,
Determining a cluster level of the data key values in response to a case where the distribution shape does not match the plurality of continuous probability distributions; And
In response to the determined cluster level being less than a predetermined threshold level, skipping conversion of the data key value
A method of performing scannable dynamic hashing comprising a. The method of claim 1,
The step of determining the conversion function,
Determining the transformation function based on a cumulative distribution function (CDF) for a distribution type of the key value distribution information
A method of performing scannable dynamic hashing comprising a. The method of claim 10,
The step of determining the conversion function,
Converting the cumulative distribution function into a set of linear functions based on a linear interpolation method
A method of performing scannable dynamic hashing comprising a. The method of claim 11,
The step of determining the conversion function,
Dividing the data key values into a plurality of groups according to the number of data key values, and calculating a linear function for each group
A method of performing scannable dynamic hashing comprising a. The method of claim 1,
The step of determining the hash index structure,
Creating a directory related to the data key value and a bucket corresponding to the directory, and mapping data key values converted according to the conversion function to the bucket
A method of performing scannable dynamic hashing comprising a. A computer-readable recording medium storing one or more computer programs including instructions for performing the method of claim 1. Determine a distribution type of the key value distribution information calculated for data key values, determine a conversion function for converting the data key values based on the result of determining the distribution type of the key value distribution information, and a predetermined conversion function A processor for converting the data key values into values corresponding to a uniform distribution by using and generating a hash index structure related to the transformed data key values; And
A memory for storing the converted data key values in the hash index structure;
An apparatus for performing scannable dynamic hashing comprising a.

Images