KR101035857B1 - Method for data management based on cluster system and system using the same - Google Patents

Abstract

In the data management method based on the cluster system providing the file replication of the present invention and the data management system using the same, a replica partition server group is allocated so that a plurality of partition servers simultaneously serve the same partition, and the replica partition server group To provide services. At this time, in the event of a node failure, data is recovered and the replica partition server group is reconfigured.

Description

METHOD FOR DATA MANAGEMENT BASED ON CLUSTER SYSTEM AND SYSTEM USING THE SAME

The present invention relates to a data management method and apparatus, and more particularly, to a data management method and system based on a distributed data management system.

The present invention is derived from a study conducted as part of the IT growth engine core technology development project of the Ministry of Knowledge Economy and the Ministry of Information and Telecommunications Research and Development. [Task management number: 2007-S-016-02, Title: Low-cost large-scale global Internet service solution Development]

With the advent of Web 2.0, Internet services are changing the paradigm from provider to user. As a result, the market for Internet services such as User Created Contents (UCC) and Personalization Services is growing rapidly.

As such, due to a paradigm shift in which Internet services are changed from provider to user, the amount of data to be managed in order to provide Internet services is rapidly increasing. Therefore, in order to provide an Internet service, efficient management of a large amount of data is required.

Recently, systems are being developed that can provide low-cost computing nodes on a large scale to provide computing power and disk space to support large amounts of data. These systems are composed of a plurality of nodes, and provide high availability using file replication technology and logging technology in case of a node failure.

Related research includes distributed file system technologies such as Google's "GFS" and Apache Software Foundation's "HDFS". This distributed file system employs replication techniques for files to provide high availability of the system.

In addition, cluster-based distributed data management systems such as "Bigtable" and "HBase" are based on distributed file systems that provide such file replication. It offers availability.

1 is a diagram illustrating a general cluster-based distributed data management system.

Referring to FIG. 1, general cluster-based distributed data management systems include one master server 12 and n partition servers 14-1, 14-2,..., 14-n.

The master server 12 performs overall load balancing and metadata management.

Each partition server 14-1, 14-2, ..., 14-n is responsible for the actual data service. Each partition server 14-1, 14-2, ..., 14-n serves only a specific part of the database. Such a system has the following characteristics, unlike previous data management systems.

First, only single operation transactions for one row are supported for change operations. Second, when inserting, deleting, or changing data, it does not change the previous data, but instead marks it as deleted or adds a new value. Third, a change operation of data is performed in an update buffer in memory, which is a certain size or is periodically written to disk. Fourth, files generated by the recording of the update buffer are periodically integrated to delete unnecessary data. Fifth, change operations performed in memory to cope with system errors are first recorded in a log file. Sixth, to deal with disk failures, we use the file replication function of distributed file system and redo technique using information stored in log files.

2 is a flowchart illustrating a management procedure of a general cluster-based distributed data management system.

Referring to FIG. 2, a request is made to a master server to perform a service request of a user (201). The master server selects a server to be serviced using partition allocation information (202), and a server selected by the master server. The service request is transmitted, and a change operation is requested to the corresponding partition server according to the service type (211). First, the change history is recorded in the log file (212), and the change is performed on the memory (213). If the information on the memory is accumulated above a certain size or a predetermined time has elapsed, check whether it is time to reflect the memory content on the disk (214). Store permanently. When data of a predetermined size or more is recorded in the disk file, the distributed file system replicates it (216), and completes the change operation (217).

The replication method can be a synchronous method that completes replication at the point of data recording, and an asynchronous operation method that performs a replication in the system by reporting that a user request is completed when one file is recorded.

The user's search operation request 221 first checks the contents in the memory buffer (222), and provides the search result, or reads the information stored in the disk (224) and provides the search results therefrom (225).

3 is a flowchart illustrating a data recovery procedure in a general cluster-based distributed data management system.

Referring to FIG. 3, if a node error is detected at the master server 12 (301),

The log file created by the node having the error is sorted according to the criteria used for partitioning and the logging recording order (302), and the log is partitioned by partition (303). After that, the redo log path information is transmitted together with the redundancy log path being allocated to the new partition server.

The partition server 14-1, 14-2, ..., 14-n reads the redo logs sequentially, applies the updates to the update buffer, and then recovers the data by writing them to disk (305). Start the data service (306).

Such a system provides scalability for large amounts of data and high availability for the data.

However, when a user's request is concentrated on specific data, the load on the node serving the partition increases, and the service cannot be performed properly. To solve this problem, we need a way to replicate the same data and service it on multiple nodes. In other words, the load should be distributed in such a way that the user-intensive data is replicated to multiple nodes to provide services from multiple nodes. To this end, existing data management systems provide a service by building a replication system and synchronizing data between the original system and the replication system.

However, in the existing data management system, a high performance is required because the corresponding system must directly handle the physical replication such as synchronization when data replication and replicated data are changed. In addition, the existing data management system only provides a table-level data replication function, causing a waste of data replication.

Accordingly, the present invention uses a distributed file system of a cluster system that already provides a physical data replication function, and thus, unlike the conventional method, a logical data replication method using files and redo logs that are already replicated in a distributed file system is provided. Through this method, a high performance data replication method can be provided, and furthermore, a data management method and system capable of providing a data replication function of a partition unit that is a part of a table can be provided.

A data management method according to an aspect of the present invention for achieving the above object, the step of allocating a replica partition server group for a plurality of partition servers to service the same partition at the same time, the service using the replica partition server group Providing and recovering data in the event of a node failure, reconfiguring the replica partition server group.

According to another aspect of the present invention, a data management system includes a replica partition server group for simultaneously serving the same partition and a master server for allocating the replica partition server group. At this time, the replica partition server group recovers data when a node error occurs, and reconstructs data in memory.

According to the present invention, a high performance data replication function is provided through a logical data replication method using replicated files and a redo log using a cluster-based distributed file system that basically provides a file replication function. In addition, unlike the conventional table providing a data replication function, it is possible to prevent the waste of data replication by providing a data replication function in a partition unit.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

4 is a flowchart illustrating a management procedure of a cluster-based distributed data management system according to an embodiment of the present invention. However, in FIG. 4, the client and the master server are distinguished based on the dotted line.

Referring to FIG. 4, when a client requests a master server to access a service performing server (S401), the master server receives a request for the access (S411), and retrieves partition allocation information (S412). Search for a set of partition servers that serve partitions.

Thereafter, the master server determines a partition server by determining an operation type from the partition allocation information (S414). At this time, if it is determined that the operation type is a change operation (S413), a primary partition server is determined in the partition server group (S418). If the operation type is determined as a search operation (S413), one partition server of the primary partition server and the secondary server (secondary partition server) is determined according to the load state of the node (S416).

The client receives the determined partition server information (402), and requests the partition server to perform a service (403).

As described above, in order for multiple partition servers to simultaneously serve the same partition, partition server allocation is required for each partition.

When the master server determines a partition server to provide a service to a client, in case of hot data, a partition server serving the same partitions is allocated as many as the number of replicated files. Here, hot data refers to data that is expected to have high access frequency. For reference, cold data refers to data that is expected to have low access frequency.

In the determined replica partition server group, one is designated as the primary partition server and the other is designated as the secondary partition server. The primary partition server is responsible for both change and retrieval services, and the secondary partition server is responsible for retrieval services only.

As for the information on the replica partition server group, the partition allocation information is managed as shown in Table 1 below.

Table Name & Partition Range Partition Service Server Primary Server T1 & from 1 to 100 Node 1, Node 2, Node 3 Node 1   T1 & from 101 to 150 Node 2 Node 2 T2 & from 1 to 200 Node 2, Node 3, Node 4 Node 3

When a user's service request comes in, the partition server to be serviced is determined by considering the service type and key information of the corresponding row.

If it is a search service and the service server is replicated at the same time, the partition server is allocated according to the existing work load distribution method (S416), and in the case of a change operation (S413), it is assigned to the primary partition server (S418). ).

The search service is serviced in the same manner as described with reference to FIG. 2, and the change operation is performed as shown in FIG. 5 below.

5 is a flowchart illustrating a service providing method for a change operation according to an embodiment of the present invention. However, in FIG. 5, the private partition server and the secondary partition server are shown based on the dotted lines.

Referring to FIG. 5, upon receiving a request for a change operation from a client (S511), the primary partition server records information in a log file as in the conventional method (S512). Thereafter, the primary partition server requests a change operation of data in memory to the secondary partition servers (S513).

When the secondary partition servers receive an in-memory change request (S521), the secondary partition servers perform a memory change operation (S522), and transmit a performance completion message for the change operation to the primary partition server (523).

When the primary partition server receives the execution completion message from the secondary partition servers (S514), the primary partition server performs a change operation on data in its memory (S515).

Data of the update buffer in the memory is reflected (or stored) on the disk if a specific condition is satisfied, and data of the update buffer is deleted at the same time. At this time, it operates differently for the partition managed by the primary partition server and the partition managed by the secondary partition server.

That is, the information is reflected on the disk only for the partition managed as the primary partition server. The process of reflecting the corresponding information on the disk (S516, S517, S518, S519) is the same as the process (S214, S215, S216, S217) of the disk described in FIG. Therefore, detailed description thereof will be omitted.

In this embodiment, the data recovery method in case of a node failure is to divide the log file information managed by the failed node into partitions and then assign each partition to new servers. It depends on whether it is secondary or not.

When the role of the server is primary, the data in the memory is recovered and reflected on the disk by using the logging information on the partition in the same way as the recovery in the existing system.

In the case where the role of the server is secondary, the change operation does not occur in the server where the error occurs, and thus no logging information exists. Therefore, the new partition server allocated as the secondary partition server reflects all changes on the data accumulated in the memory up to now from the primary partition server serving the partition.

6 is a flowchart illustrating a data recovery method in the event of a node failure in a data management system according to an embodiment of the present invention.

Referring to FIG. 6, the master server divides the log file managed by the node in which the error occurs by partition (S601), and selects nodes to service each partition (S602). Thereafter, it is determined what the role of the node where the error occurs for each partition (S603). At this time, when the node in which the error occurs is determined to be the primary role, the partition log information is transmitted to the primary partition server (S604), and the newly selected node recovers data for the corresponding partition (S605). If the failed node is determined to be a secondary role, information about the primary partition server of the corresponding partition is transmitted to the secondary partition server (S606), and data recovery 607 is performed as the secondary partition server. .

7 is a flowchart illustrating a method of recovering data as a secondary partition server according to an embodiment of the present invention.

Referring to FIG. 7, first, the secondary partition server requests logging information recorded so far from the primary partition server (S701).

The primary partition server extracts only logging information belonging to the partition from the logging file (702), and passes the extracted logging information back to the secondary partition server (703). At this time, the primary partition server continues to service (S704).

The secondary partition server receives the logging information file from the primary partition server (705) and reconstructs the data in the memory (S706). When the reconstruction of the data in the memory is completed, the primary partition server notifying the completion of the reconstruction is notified to the primary partition server (S707).

When the primary partition server receives the completion message (708), the primary partition server collects only the logging information added by the change operation after transferring the logging information file and re-delivers it to the secondary partition server (S709). At this time, the primary partition server stops the service until it receives a completion notification from the secondary partition server (714) (S710).

The secondary partition server receives the added logging information from the primary partition server again (S711), reconstructs the added logging information in the memory, and matches the added logging information with the data in the memory (S712).

When the matching between the reconstructed added logging information in the memory and the data in the memory is completed, a second completion message indicating completion is notified to the primary partition server side (713).

When the primary partition server is notified of the second reconfiguration complete message from the secondary partition server (S714), the service is resumed by the new second partition server servicing the corresponding partition (S715).

On the other hand, as described above, restoring information in the memory in two steps is to minimize service downtime by the primary partition server during the recovery.

As described above, the data management method of the present invention, unlike the conventional method, provides a logical data replication method using files already replicated in a distributed file system and redo logging information.

Furthermore, in the present invention, by copying data in a smaller partition unit than in a table unit, unlike the conventional copying in a table unit, data is copied in a partition unit, thereby minimizing waste due to data replication.

1 is a diagram illustrating a general cluster-based distributed data management system.

2 is a flowchart illustrating a management procedure of a general cluster-based distributed data management system.

3 is a flowchart illustrating a data recovery procedure in a general cluster-based distributed data management system.

4 is a flowchart illustrating a management procedure of a cluster-based distributed data management system according to an embodiment of the present invention.

5 is a flowchart illustrating a service providing method for a change operation according to an embodiment of the present invention.

6 is a flowchart illustrating a data recovery method in the event of a node failure in a data management system according to an embodiment of the present invention.

7 is a flowchart illustrating a method of recovering data as a secondary partition server according to an embodiment of the present invention.

Claims (10)

In the cluster system-based data management method that provides file replication, Allocating a replica partition server group so that a plurality of partition servers simultaneously serve the same partition; Providing a service using the replica partition server group; and Reconstructing the replica partition server group by recovering data in case of node failure Cluster system based data management method comprising a. The method of claim 1, wherein the step of assigning a replica partition server group, Allocating a partition server serving the same partition by the number of replicated files; And Determining whether a role of the assigned partition server is a primary partition server that is in charge of both a change service and a search service or a secondary partition server that is responsible only for the search service Cluster system-based data management method comprising a. The method of claim 2, wherein the providing of the service by using a replica partition server group includes: When providing the change service, storing the same data information through the change operations of the memories of all the secondary partition servers belonging to the replica partition server group; And Reflecting only the partition provided with the service by the primary partition server on the disk Data management method comprising a. The method of claim 1, wherein providing a service using the replica partition server group includes: And selecting a partition server in consideration of a service type as well as key value information of a corresponding row of a user's service request, and providing the service. The method of claim 2, wherein the reconstructing the replica partition server group by recovering data in the event of a node failure comprises: The data recovery procedure by the primary partition server and the data recovery procedure by the secondary partition server are different. The data recovery procedure by the secondary partition server is performed in a two-step procedure, The procedure of the two steps, Reconfiguring memory in the secondary partition server based on a logging information file from the primary partition server; And Reconstructing the data in the memory of the secondary partition server based on the logging information file added from the primary partition server to match the data in the memory with the latest data. A data management system based on cluster system that provides file replication, A replica partition server group serving the same partition at the same time; And A master server for allocating the replica partition server group; And said replica partition server group recovers data in the event of a node failure and reconstructs data in memory. 8. The server of claim 7, wherein the replica partition server group is configured by the master server. A data management system, which is designated as a primary partition server providing both change service and search service, and a secondary partition server providing only search service. The method of claim 8, When the node error occurs, in the case of the primary partition server, after recovering the data in the memory by using the logging information for the partition, and reflecting it on the disk. The method of claim 8, When the node error occurs, in the case of the secondary partition server, the data management system reflects all changes of data accumulated in the memory up to now from the primary partition server serving the partition.

Images