KR20150045073A - Data Operating Method And System supporting the same - Google Patents

Abstract

The present invention relates to a data operating method. The present invention provides a hybrid data operating method collecting the advantages of cuckoo hashing and chained hashing, and to a system supporting the same. In the present invention, disclosed are a key-value store system which includes a storage device which stores a cuckoo hash table and a chained hash table and a control device which previously inspects the entry of the cuckoo hash table in a data operating process and applies the chained hash table when the cuckoo hash table is not applied or an overhead is generated in an application process; and a data operating method thereof.

Description

DATA OPERATING METHOD AND SYSTEM SUPPORTING THE SAME

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to data storage, and more particularly, to a hybrid data management method that combines the advantages of cache and hashing.

Computing tasks are based on collecting and storing a lot of data. The computing operation includes an operation of detecting and providing stored data when a specific data retrieval request occurs. In this process, it is possible to repeatedly and frequently occur situations in which a large amount of data is stored and a stored data is read. Accordingly, the task of efficiently storing and reading data in a computing task is an important task. Various data storage methods have conventionally been proposed for such data operation.

For example, conventionally, a data operation method based on a chained hash table or chaining or separate chaining has been used. Chained Hashtable is a data structure that is mainly used for key management of distributed memory based on NoSQL database. It has a feature of linking collided keys by using linked list. Such a chained hash table has a feature that the insertion speed of a key is fast because it only adds the list head. However, the chained hash table has a drawback in that the read speed is lowered because the entire list must be scanned to find the key. This chain hash table has the disadvantage that it can not guarantee the worst case processing time theoretically.

To overcome the drawbacks of the above-described chain hash table, a Mercury method has been proposed. Mercurial is an open source implementation of Memcached, and existing Memcached uses a global lock to issue a race condition when multiple threads access a hash table. Mercury uses fine-grained locks to increase performance in multithreaded environments. In other words, the Mercury method uses each lock for each table bucket (entry). However, this Mercury method improves the performance of the chain hash table in a multithreaded environment, but it does not overcome the inherent drawbacks of the chain hash table that list scans are needed.

Accordingly, it is an object of the present invention to provide a data management method and a system for supporting the same, which can overcome the disadvantages of data management in various data structures.

According to an aspect of the present invention, there is provided a storage apparatus for storing a cookie hash table and a chain hash table, an entry check of a cookie hash table at the time of data operation, A key-value store system including a control apparatus for performing application of the above-described chain hash table is disclosed.

The control device may perform an entry check of the cookie hash table when requesting data insertion, and to control the entry to add the corresponding data to the list of the hashed hash tables when the data is not empty.

The control device can check whether the system enters a set condition and control to move data added to the chain hash table list to the cook hash table.

The control device can control to apply the two hash functions applied to the cookie hash table as the same function.

The control device may add a field for storing an index to an entry of the cookie hash table when writing the data to the entry of the cookie hash table.

The control unit can control the cache hash table to be searched in advance and the failed hash table search to be performed in case of a failure.

The control unit moves the entry of the corresponding data to the head of the list when the data in the chain hash table is referenced and controls to move the corresponding data to the cook hash table when the data moved to the list head is additionally referenced can do.

The present invention also includes a process of receiving a data insertion request, performing an entry check of a cookie hash table, and inserting the data into a list of a chain hash table when the entry of the cookie hash table is filled A configuration of a data operation method is disclosed.

The method may further include a step of entering the system into a predetermined condition, and a step of moving data added to the chain hash table list according to the entry into the cookie hash table.

The method may further include adding a field for storing an index to an entry of the cookie hash table upon writing the data into the entry of the cookie hash table.

The entry process may be a process in which the system enters an idle state.

The present invention also provides a data operation method comprising the steps of receiving a data retrieval request, preceding an entry in a cookie hash table for the data, and retrieving a list of chained hash tables in case of an entry retrieval failure .

The method may further include the step of operating the hash function as a function in the searching of the cookie hash table.

The method may further include moving an entry of the data to the head of the list when the data in the hash table is referenced.

The method may further include moving the corresponding data to the cookie hash table if data moved to the list head is further referenced.

The foregoing is a somewhat broad description of features and technical advantages of various embodiments in order to enable those of ordinary skill in the art to better understand the various embodiments of the present disclosure described below. Additional features and advantages of various embodiments that form the subject matter of the claims of this disclosure, in addition to those features and advantages, will be better understood from the following detailed description.

As described above, according to the data operation method proposed by various embodiments of the present invention and the system supporting the same, the present invention improves data reading and writing speed through improved data operation, .

FIG. 1 is a schematic view of a system to which a hybrid hashing operation according to an embodiment of the present invention is applied.
2 is a block diagram showing a system configuration of the present invention.
FIG. 3 is a detailed block diagram of a control apparatus of a system according to an embodiment of the present invention.
4 is a flowchart illustrating a method of inserting data by hybrid hashing according to an embodiment of the present invention.
5 is a diagram for explaining data insertion by hybrid hashing.
FIG. 6 is a flowchart illustrating a method of searching for data by hybrid hashing according to an embodiment of the present invention.
7 is a diagram for explaining data retrieval by hybrid hashing.
FIG. 8 is a diagram showing a successful search result of 100,000 to 1 million of the results of measurement of work time according to the use of the hybrid hash table and the hybrid hash table of the present invention.
FIG. 9 is a diagram illustrating a result of failure detection for 100 to 1 million of the results of the operation time measurement according to the use of the hybrid hash table and the hybrid hash table of the present invention.
FIG. 10 is a diagram showing search results having a 10% reference locality of 100,000 to 1 million of the results of measurement of work time according to the use of the hybrid hash table and the hybrid hash table of the present invention.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that, in the drawings, the same components are denoted by the same reference symbols as possible. Further, the detailed description of known functions and configurations that may obscure the gist of the present disclosure will be omitted. In other words, it should be noted that only the parts necessary for understanding the operation according to the embodiment of the present disclosure will be described in the following description, and the description of the other parts will be omitted to the extent that the spirit of the present disclosure is not disturbed.

FIG. 1 is a schematic view showing the appearance of a system to which hybrid hashing operation according to an embodiment of the present invention is applied, and FIG. 2 is a block diagram showing a system configuration of the present invention.

1 and 2, a system, for example, a key-value store system 10, to which the hybrid hashing of the present invention is applied, may include at least one storage device 150 and a controller 160.

The key-value store system 10 according to the present invention having the above-described configuration can store data in the memory area of the storage device 150 or store data in the memory area of the storage device 150 in accordance with a hybrid hashing method of operating a cookie- It can support operation. That is, the key-value store system 10 performs data processing based on a cuckoo hash table-based cuckoo hashing, but additionally performs chained hashing based on a chained hash table, To perform data processing on the basis of the data. The key-value store system 10 enhances data retrieval efficiency through cache hashing and supports data insertion speed through chain hashing. Accordingly, the key-value store system 10 of the present invention can ensure a fast data insertion speed while supporting a faster search time.

The storage device 150 may store processing data of the control device 160. [ The storage device 150 may store various data according to the type of service supported by the corresponding key-value store system 10 or according to the data collection type of the operator. For example, the storage device 150 may temporarily store data (e.g., telephone numbers, messages, audio, media content (e.g., music files, video files), applications, etc.). The storage device 150 may also store various types of big data. For example, the storage device 150 may store various data such as user location data of at least one mobile communication device, user's web use data, user's call data, user's purchase data, and the like. As described above, the key-value store system 10 according to the embodiment of the present invention is not limited to the types and characteristics of data stored in the storage device 150. [

Such a storage device 150 may include a cookie hash table and a chain hash table for storing data. The cookie hash table can be a data structure in which data is inserted and retrieved by the cookie hashing method. A chain hash table can be a data structure in which data is inserted and retrieved in a certain area of the list according to a chain method. Particularly, the storage device 150 of the present invention can provide a cookie hash table as a basis in a process of storing data, but can provide a chain hash table as an auxiliary use.

The storage device 150 may be a flash memory type, a hard disk type, a micro type, and a card type (e.g., an SD card (Secure Digital Card) or an XD card eXtream Digital Card), a random access memory (RAM), a static random access memory (SRAM), a read only memory (ROM), a programmable ROM (PROM), an electrically erasable programmable read- A storage medium of at least one type of magnetic random access memory (MRAM), magnetic random access memory (Magnetic RAM), magnetic disk, and optical disk type memory. The key-value store system 10 may operate in association with a web storage that performs storage functions of the storage device 150 over the Internet.

The controller 160 may perform processing and transfer of various signals related to the control of the key-value store system 10 and transmission and processing of data. For example, the control device 160 can perform control such as data collection and insertion of data into the storage device 150, retrieval of data stored in the storage device 150, deletion or update of data stored in the storage device 150 have. The control device 160 of the present invention can operate the cookie hashing method and the chained hashing method on the data stored in the storage device 150 in the process of inserting and retrieving data. According to one embodiment, the control device 160 can perform insertion of specific data based on the cookie hashing method. In addition, the controller 160 may control the chain hashing method to be applied when insertion overhead occurs when data is inserted. When the key-value store system 10 is in a set condition, for example, in the "Idle" state, the controller 160 can switch the data applied in the chain-hashing method according to the KuKo hashing method. Such a control device 160 may include a configuration as shown in FIG.

The embodiments of the present invention as described above may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable recording medium. The computer-readable recording medium may include a program command, a data file, a data structure, and the like, alone or in combination. The program instructions recorded on the recording medium may be those specially designed and constructed for various embodiments of the present invention or may be available to those skilled in the art of computer software.

In addition, in the key-value store system 10, the controller 160 is connected to various configurations such as a communication unit, an input unit, and a display unit according to the purpose of use of the corresponding system, , A display function, and the like. In addition, the control device 160 in the key-value store system 10 can be further connected to a camera or a sensor module, and various user functions according to predefined program routines based on image information or sensor information provided by the corresponding configurations .

The key-value store system 10 of the present invention described above can be implemented as a server-client system using the control device 160 as a server and the storage device 150 as a client. Also, the key-value store system 10 of the present invention can be implemented as an integrated server device in which a plurality of storage devices 150 directly accessible by the control device 160 are integrated. Alternatively, the key-value store system 10 of the present invention may be implemented as one computing device in which the controller 160 and the storage device 150 are physically located within a certain space. For example, the key-value store system 10 of the present invention may be implemented as a computing device in which a central processing unit and a memory device are integrated on a board.

FIG. 3 is a detailed block diagram of a control apparatus of a system according to an embodiment of the present invention.

Referring to FIG. 3, the controller 160 of the present invention includes a cookie hashing processor 161 and a chain hashing processor 163, and can perform a hybrid hashing process on data based on the cookie hashing processor 161 and the chain hashing processor 163.

The cookie hashing processor 161 can perform the cookie hashing process. Cuckoo hashing can use two hash tables and two hash functions for each table. A key has one entry for each table using two hash functions. If the key exists in a hash table, the key must be in one of the two entries. Such a cache hashing process can provide fast and reliable retrieval, deletion, and update operations, unlike techniques such as separate chaining or linear probing, in which the retrieval time is variable, since only two locations need to be checked for key retrieval.

When inserting data into the cookie hash table, the cookie hash processor 161 of the present invention inserts an empty entry in two entries found through the hash function. If both entries are stored with different keys, the cookie hash processor 161 may call the chain hash processor 163. And the cookie hash processor 161 may forward the key store to the chain hash processor 163.

On the other hand, the cookie hashing processor 161 can receive the key transmitted to the chain hash processor 163 again when the key-value store system 10 is in the "Idle" state. The cookie hashing processor 161 can store the received key according to the cookie hashing method. When inserting data into a cookie hash table in a state where all other keys are stored, the cookie hashing processor 161 selects one of the two entries to move the existing key to another place, Insert the key in place. Since each key is located in one of the two entries, the moved key is moved to its own entry in the other table, and the cookie hashing processor 161 can repeat the above-described key insertion process until it finds an empty entry .

The chain dehashing processor 163 may perform chain hash table management. When the key store is received from the cookie hashing processor 161, the chain dehashing processor 163 may perform list processing for storing the key. And the chain de-hashing processor 163 may pass the key added to the list to the cookie hash processor 161 if the key-value store system 10 is in the "Idle" state.

FIG. 4 is a flowchart illustrating a method of inserting data by hybrid hashing according to an embodiment of the present invention, and FIG. 5 is a view for explaining inserting data by hybrid hashing.

Referring to FIGS. 4 and 5, in the process of inserting the key K by the hybrid hashing, the control device 160 may calculate an entry using the hash function according to the cookie hashing method in step S101. At this time, the control device 160 can calculate two entries h1 (k) and h2 (k) using two hash functions in one key. When the entry is calculated, the controller 160 can check whether the entry calculated in step S103 is empty. On the other hand, before the step S101, the control device 160 may reduce the unnecessary load on the hash join by applying a Bloom filter.

If h1 (k) or h2 (k) is empty in step S103, the control device 160 can insert the key K into the empty slot in step S105. On the other hand, if h1 (k) and h2 (k) are not empty in step S103, the control device 160 adds the key K to the h (K) list of the chain hash table in step S107.

Thereafter, the controller 160 can check whether the key-value store system 10 is in a specific setting condition in step S109. For example, the control device 160 can check if the key-value store system 10 is in the "Idle" state. Here, if the control device 160 is not in the "Idle" state, the control device 160 may skip step S111. Thereafter, the control device 160 performs step S109 of checking whether the key-value store system 10 is in the idle state in a predetermined period or in real time, and thereby controlling the execution of step S111.

On the other hand, if the key-value store system 10 is in the set condition in step S109, the controller 160 controls the key inserted in the hashed chain table in step S107 to move the key to the cookie hash table in step S111 . The key movement process of the cookie hash table can be handled according to the cookie hashing inherent method. In this process, the keys in the chained hash table are slowly moved by a separate thread. Through this process, the control device 160 can delay the "pushing out " process, which creates a large overhead in the cookie hashing insert operation, and process the response so that the client can respond quickly.

Through the above-described process, the key-value store system 10 of the present invention minimizes the insertion cost of the cookie hash table at a specific time according to the occurrence of a collision in the key insertion process using a chain hash table, Thereby improving the system response characteristics.

FIG. 6 is a flowchart for explaining a method of searching for data by hybrid hashing according to an embodiment of the present invention, and FIG. 7 is a view for explaining data searching by hybrid hashing.

Referring to FIGS. 6 and 7, in the method of searching for data by hybrid hashing according to the present invention, when the key k search is requested in step S201, the controller 160 applies a cookie hashing method in step S203, (K) and h2 (K) according to the application of the hash function to the hash function. On the other hand, before step S201 or step S206, the controller 160 may reduce the unnecessary load on the hash join by applying a Bloom filter.

The control device 160 checks whether or not the entry is empty in step S205. In step S205, if the entry is empty, the control device 160 performs a search failure process and returns. Since the cache hash table is used first in the data insertion process, if the entry is empty, h (K) of the chained hash table is also guaranteed to be empty. However, in order to do this, it is possible to perform the deletion operation of the h (k) of the chained hash table when the key is deleted.

If data is recorded in the entry, the control device 160 can confirm whether the data corresponds to the key to be searched in step S207. If the data retrieval for the corresponding key is successful in step S207, the process returns.

On the other hand, if data is recorded in the entry but is not data for the corresponding key, the controller 160 additionally searches the list in the h (K) of the chain hash table in step S209. The control device 160 then returns if the key retrieval is successful in the chain hash table. Here, when the key in the chain hash table is referred to, the controller 160 moves the entry of the key to the head of the list in order to increase the retrieval speed of the frequently referenced key. Then, when the key moved to the list head is once again referred to, the controller 160 moves the key to the cookie hash table.

The data update and deletion process according to the hybrid hashing operation of the present invention may basically be the same as the search process. For example, the control device 160 may perform a location search using a corresponding key-value to be updated. At this time, the control device 160 may preferentially search for the cookie hash table as described above, perform location verification of the key, and update the value of the key. Here, if the corresponding key does not exist in the cookie hash table, the controller 160 can search for the chained hash table. Even in the deletion process, the control device 160 can first search the cookie hash table, and control to retrieve the chained hash table as needed.

As described above, in the key-value store system 10 according to the embodiment of the present invention, when there is a key in the cookie hash table, since success is guaranteed only by a maximum of two references, separate chaining can not guarantee the worst case It is more stable.

Meanwhile, the key-value store system 10 of the present invention can apply an additional method to improve the iterative calculation of the hash function and to enhance the response delay in the case of a search failure. To explain this in more detail, the control device 160 uses the same hash function for h1 (K) and h (K). The controller 160 may calculate an index of the chain hash table by calculating modulo h1 (K) only through mod operation. In addition, the control device 160 ensures that h1 (K) is empty and h2 (K) is not used in the cookie hash table, so that h2 (K) is immediately returned without finding h2 . In addition, the key-value store system of the present invention reduces the hash function calculation by adding a field for storing an index to the entry of the cookie hash table. This field stores the index of h2 (K) when the key is stored in the position of h1 (K), and the index of h1 (K) when it is stored in the position of h2 (K). Implementations use the same space as the next pointer to the next entry in the list of chained hash tables, using union {}, thus eliminating additional memory usage.

FIGS. 8 to 10 are diagrams showing the results of measurement of work time according to the use of the hybrid hash table and the hybrid hash table of the present invention. In particular, we use the chain hash table and the hybrid hash table to measure the time taken to perform a lookup operation with 40% of the table entries being used. For 100,000 to 1 million cases, Figure 9 shows a failed search, and Figure 10 shows a search result with 10% reference locality.

Experimental results show that about 90% of Kigaku hash tables are stored. Hybrid hashing methods have an average of 7.4%, 15% and 13%, respectively, for successful search and 13% and 24%, respectively, compared to the chain hashing method. On the other hand, unsuccessful searches have an average slow rate of 4.6% and up to 12%, because if they fail, they will have to search both the cookie hash table and the chained hash table entries. However, even if the retrieval fails, the proposed method stores only one key in the list of chained hash tables as shown in Table 1.

hashing successful lookup unsuccessful lookup average worst average worst Chaining 1.17 7 1.26 7 Hybrid 1.13 2 1.37 3

In the worst case, the retrieval time can not be measured accurately because the retrieval time is very short. However, it is expected that the worst case retrieval time is shorter than the hybrid hash table.

In the present invention, a technique for improving the search performance of a single node in the key-value store system 10 is proposed. The present invention applies delayed writing using a chain hash table together to compensate for the slow insertion speed of the cookie hashing while using fast and stable cookie hashing. The search speed of the present invention is up to 24% faster when there is reference locality.

In the above description, the hybrid hashing method of the present invention is described as being implemented in one system, but the present invention is not limited thereto. That is, the hybrid hashing scheme described in the present invention can be basically applied to a key-value store scheme as described above. The key-value store approach would be to distribute keys across multiple servers and provide storage services through fast read / write in memory. As a result, the hybrid hashing method of the present invention can be applied to a server apparatus corresponding to a plurality of systems in the same form. In particular, the server apparatus implemented by applying the hybrid hashing scheme of the present invention assists efficient use of memory based on the cache hashing, advantage of implementation, guarantee of constant time work even in the worst case, And can support improved insertion and retrieval speed improvements.

And the embodiments of the present disclosure disclosed in this specification and drawings are merely illustrative of specific examples for the purpose of facilitating the understanding of the disclosure of the present disclosure and are not intended to limit the scope of the present disclosure. Accordingly, the scope of the present disclosure should be construed as being included in the scope of the present disclosure in addition to the embodiments disclosed herein, all changes or modifications derived from the technical idea of the present disclosure.

10: key-value store system
150: Storage device
160: Control device
161: Cucumis hashing processor
163: Chained hashing handler

Claims (15)

A storage device for storing the cookie hash table and the chained hash table;
A control device for performing an entry check of a cookie hash table at the time of data operation and performing the application of the chain hash table when an overhead occurs in a case where the cookie hash table can not be applied or applied;
A key-value store system, The method according to claim 1,
The control device
The key-value store system according to claim 1, wherein the key-value storage system is configured to check an entry of the cookie hash table when the data is inserted, and to add the data to the list of the chain hash table if the data is not empty. 3. The method of claim 2,
The control device
A key-value store system which checks whether the system enters a set condition, and controls to move data added to the chain hash table list to the cookie hash table. The method according to claim 1,
The control device
A key-value store system that controls to apply two hash functions applied to the cookie hash table to the same function. The method according to claim 1,
The control device
A key-value store system for adding a field for storing an index to an entry of a cookie hash table upon writing the data to an entry of the cookie hash table. The method according to claim 1,
The control device
A key-value store system for searching for a cache hash table in advance when searching for data, and performing a search for the hash table in case of failure. The method according to claim 6,
The control device
If the data in the chain hash table is referred to, moves the entry of the corresponding data to the head of the list, and if the data moved to the list head is further referenced, controls key-value store system. Receiving a data insertion request;
Performing an entry check of a cookie hash table;
Inserting the data into the list of the chain hash table when the entry of the cookie hash table is filled;
Lt; / RTI > 9. The method of claim 8,
An entry process in which the system enters a predetermined condition;
Moving the data added to the chain hash table list to the cookie hash table according to the entry;
≪ / RTI > 9. The method of claim 8,
Adding a field for storing an index to an entry of a cookie hash table upon writing the data into an entry of the cookie hash table;
≪ / RTI > 11. The method of claim 10,
The entry process
And the system enters an idle state. Receiving a data retrieval request;
A step of preceding an entry search of the cookie hash table with respect to the data and retrieving a list of the chained hash table in case of an entry search failure;
Lt; / RTI > 13. The method of claim 12,
Operating the hash function as the same function in the retrieval process of the cookie hash table;
≪ / RTI > 13. The method of claim 12,
Moving the entry of the data to the head of the list if the data in the chain hash table is referenced;
≪ / RTI > 13. The method of claim 12,
If the data moved to the list head is further referenced, moving the corresponding data to the cookie hash table;
≪ / RTI >

Images