KR101259557B1 - Cluster data management system and method for data recovery using parallel processing in cluster data management system - Google Patents

Cluster data management system and method for data recovery using parallel processing in cluster data management system Download PDF

Info

Publication number
KR101259557B1
KR101259557B1 KR1020090024150A KR20090024150A KR101259557B1 KR 101259557 B1 KR101259557 B1 KR 101259557B1 KR 1020090024150 A KR1020090024150 A KR 1020090024150A KR 20090024150 A KR20090024150 A KR 20090024150A KR 101259557 B1 KR101259557 B1 KR 101259557B1
Authority
KR
South Korea
Prior art keywords
partition
server
log
management system
data management
Prior art date
Application number
KR1020090024150A
Other languages
Korean (ko)
Other versions
KR20100070968A (en
Inventor
이훈순
이미영
Original Assignee
한국전자통신연구원
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to KR20080129637 priority Critical
Priority to KR1020080129637 priority
Application filed by 한국전자통신연구원 filed Critical 한국전자통신연구원
Publication of KR20100070968A publication Critical patent/KR20100070968A/en
Application granted granted Critical
Publication of KR101259557B1 publication Critical patent/KR101259557B1/en

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/20Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2023Failover techniques
    • G06F11/2028Failover techniques eliminating a faulty processor or activating a spare
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • G06F11/1402Saving, restoring, recovering or retrying
    • G06F11/1471Saving, restoring, recovering or retrying involving logging of persistent data for recovery
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/20Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2023Failover techniques
    • G06F11/2025Failover techniques using centralised failover control functionality
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/20Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2023Failover techniques
    • G06F11/203Failover techniques using migration
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/20Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2035Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant without idle spare hardware
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16Error detection or correction of the data by redundancy in hardware
    • G06F11/20Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2046Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant where the redundant components share persistent storage

Abstract

The present invention provides a data recovery method using parallel processing in a cluster data management system, comprising: sorting a redo log generated by a partition server in which an error occurs; Dividing the sorted redo logs by column of the partition; And restoring data on the basis of the partitioned redo log.
Cluster, data management system, partition, column, parallelism

Description

Data recovery method using parallel processing in cluster data management system and cluster data management system {CLUSTER DATA MANAGEMENT SYSTEM AND METHOD FOR DATA RECOVERY USING PARALLEL PROCESSING IN CLUSTER DATA MANAGEMENT SYSTEM}

The present invention relates to a data recovery method in a cluster data management system. More specifically, when a computing node constituting a cluster data management system fails, data is recovered using parallel processing based on a redo log created by the computing node. It is about how to.

The present invention is derived from the research conducted as part of the IT growth engine core technology development project of the Ministry of Knowledge Economy and the Ministry of Information and Telecommunication Research and Development. [Task Management Number: 2007-S-016-02] Development].

Recently, with the advent of Web 2.0, the internet service market such as UCC and personalization service is rapidly increasing as the paradigm of internet service is shifted from provider to user. This paradigm shift is rapidly increasing the amount of data that must be managed to provide Internet services. Therefore, in order to provide an Internet service, it is necessary to efficiently manage a large amount of data. However, the amount of such data is huge, so it is difficult to efficiently manage the existing database management system (DBMS) in terms of performance and cost.

In recent years, research has been conducted to improve computing performance by connecting low-cost computing nodes as a countermeasure and to improve the high performance and high availability by using software.

Bigtable and HBase are the researches for this cluster data management system. Bigtable is created by Google and applied to various Internet services of Google. HBase is an open source project that mimics Google's concept of Bigtable by the Apache Software Foundation.

FIG. 1 is a diagram illustrating a general cluster data management system, and FIG. 2 is a diagram for describing a data storage and service model of FIG. 1.

First, referring to FIG. 1, a general cluster data management system includes a master server 11 and n partition servers 12-1, 12-2,..., 12-n.

The master server 11 is in charge of overall control of the operation of the system.

Each partition server 12-1, 12-2, ..., 12-n is in charge of service for actual data.

The cluster data management system 10 uses the distributed file system 20 to permanently store logs and data.

The cluster data management system 10 has the following characteristics unlike previous data management systems in order to optimally use the use of computing resources in handling user requirements.

First, while most traditional data management systems store row-oriented data, the cluster data management system 10 is described with reference to FIG. 2, where columns (or column groups, eg, C1, Column-oriented storage based on C2, ..., Cr, Cs, Ct, ... Cn). A column group is a grouping of columns that are likely to be accessed together. Throughout this specification, the term column is used to collectively mean columns and column groups.

Second, when a change to the data occurs due to an insert or delete request, the data is stored in a form in which data having a new value is added instead of the change in the previous data.

Third, separate update buffers are provided for each row to manage data in memory. This update buffer is of a certain size or is periodically written to disk.

Fourth, in order to cope with errors, logs for re-execution are related to changes per partition server (node) where accessible from all computing nodes.

Fifth, service responsibility for data that is the target of service is divided among multiple nodes to enable service for multiple data at the same time. In addition to dividing the data vertically for column-based storage, it also divides the data horizontally to have a certain size. Hereinafter, for convenience of description, data divided horizontally to have a certain size is called a partition. One partition consists of one or more rows, and one node is responsible for servicing multiple partitions.

Sixth, unlike traditional data management systems, the cluster data management system 10 does not consider disk failure. The response to the disk error uses the file replication function of the distributed file system 20.

Low cost computing nodes can be easily brought down because there is little coping with errors using hardware. Therefore, effectively dealing with node failures at the software level can be an important factor in achieving high availability. In the cluster data management system 10, when an error occurs in a computing node, the data is restored to a state before an error occurs by using an update log recorded for use in error recovery at the node where the error occurs.

3 is a flowchart illustrating a method of recovering data in a general cluster data management system.

Referring to FIG. 3, first, the master server 11 detects whether an error has occurred in each partition server 12-1, 12-2, ..., 12-n (S310).

If an error is detected, the master server 11 sorts the redo log created by the partition server (eg, 12-1) in which the error occurred in ascending order based on predetermined reference information, such as a table, a row key, and a log serial number. (S320).

In order to reduce the number of disk seeks during data recovery based on the rerun logs in each partition server, the rerun logs are divided by partitions (S330).

A plurality of partitions that have been in service at the failed partition server 12-1 are allocated to be in service at new partition servers (eg, 12-2, 12-3, and 12-5) (S340).

At this time, redo log path information about the partition is also transmitted.

Each assigned partition server 12-2, 12-3, and 12-5 sequentially reads the redo log, reflects the update to the line buffer, and then recovers the data by writing to the disk (S350).

When data recovery is completed in parallel by each partition server 12-2, 12-3, and 12-5 allocated in this manner, each partition server 12-2, 12-3, and 12-5 is restored partition. Restart the data service for (S360).

In this method, data recovery is processed in parallel by partitioning partitions that were being serviced by a failed partition server 12-1 into multiple partition servers 12-2, 12-3, and 12-5. It can be done.

However, when each partition server (12-2, 12-3, 12-5) recovering partitions according to log files divided by partitions has multiple CPUs, each partition server (12-2, 12-3, 12) -5) have the disadvantage that they do not utilize the CPU resources they own properly, and also has a disadvantage that they do not make good use of the data storage model that physically divides and stores each column when storing data.

The present invention was created in view of the above problems, and when an error occurs in a partition server in a cluster data management system, the master server detecting the partition server divides a redo log created by the partition server by column of a partition. The purpose is to provide a data recovery method in a cluster data management system in which partitions are allocated to other partition servers, and the assigned partition servers can recover data based on partitioned redo logs.

Another object of the present invention is a data recovery method in a cluster-based data management system capable of recovering data in parallel by utilizing a plurality of CPU resources held in the partition server at the time of data recovery based on the partitioned redo log In providing.

In order to achieve the above object, in accordance with an aspect of the present invention, a data recovery method using parallel processing in a cluster data management system comprises the steps of: sorting the redo log created in the partition server in error; The redo logs sorted based on preset partition configuration information are classified by partitions that the partition server is in service, and then sorted again by the columns of each partition, and a file in which the redo logs classified by the columns is recorded. Dividing by column; And selecting a partition server to service partitions in service in the failed partition server, allocating a partition served by the failed partition server to the selected partition server, and a path to a file partitioned by the column. And reconstructing the partition assigned based on a log recorded in the file divided by the column corresponding to the path information by transferring information, and rebuilding the partition. The method may include: generating, by the selected partition server, a thread corresponding to the file partitioned by the row, and recovering data based on a log in which the generated thread is recorded in the file partitioned by the row. It features.

According to another aspect of the present invention, a cluster data management system for recovering data by using parallel processing is a partition server and a partition server for servicing at least one or more partitions and recording a redo log according to the service of the partition. In the event of an error, the redo logs sorted based on the predetermined partition configuration information are sorted by the partitions, and then are sorted by the columns of the partitions, and the files containing the redo logs sorted by the columns are partitioned by the columns. Selecting the partition server to rebuild the partition based on the redo log, the master server for allocating the partition and delivers the path information for the file divided by column, the partition server is the path The mas based on the log recorded in the divided file corresponding to the information Rebuilding the partition allocated from the server, creating a thread corresponding to the partitioned file, and recovering data in parallel based on the redo logs recorded in the file partitioned through the created thread. It features.

According to the present invention, by redistributing the redo log for each column of the partition, the I / O of the disk can be reduced when the redo log access is partitioned for data recovery.

In addition, in recovering data based on the partitioned redo log, the partition server utilizes a plurality of CPU resources held in itself to increase its own resource utilization and to simultaneously execute threads operating on multiple CPU resources. The data can be recovered in a parallel manner.

In particular, there is an advantage of reducing the time required to provide a normal data service by reducing disk I / O of a partitioned redo log and restoring data in a parallel processing method using a plurality of CPUs.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in detail, focusing on the parts necessary to understand the operation and action according to the present invention.

On the other hand, the term 'column' is used throughout the specification to refer to the term 'column' and 'column group'.

Hereinafter, with reference to the accompanying drawings will be described in detail with respect to the present invention.

The partition server proposed in the present invention, that is, a node error recovery method, uses a characteristic in which data is physically divided and stored for each column.

4 is a block diagram illustrating an error recovery method in the cluster data management system according to the present invention, FIG. 5 is a diagram for explaining the arrangement of a redo log of a master server, and FIG. 6 is a redo log of a master server. It is a figure for demonstrating division | segmentation.

First, referring to FIG. 4, the cluster-based data management system includes a master server 100 and n partition servers 200-1, 200-2,..., 200-n.

The master server 100 controls each partition server (200-1,200-2, ..., 200-n), and whether an error occurs in each partition server (200-1,200-2, ..., 200-n) If a failure occurs, partition the redo log created by the failed partition server by column of partitions, and rebuild partitions that were in service on the failed partition server using the partitioned redo log. Choose a new partition server to use.

For example, when an error occurs in the partition server 200-3, the master server 100 sorts the rerun log generated by the partition server 200-3 in ascending order based on predetermined reference information.

The preset reference information includes a table, a row key, a column, and a log sequence number (LSN).

The master server 100 divides the sorted redo log by the columns (eg, C1 and C2) of the partitions (eg, P1 and P2) that were in service in the partition server 200-3 that failed. After P1.C1.LOG, P1.C2.LOG, P2.C1.LOG, P2.C2.LOG), the partitions P1 and P2 that were serviced by the failed partition server 200-3 are newly renewed. Select partition servers (eg, 200-1, 200-2) to service.

The master server 100 allocates partitions P1 and P2 to the selected partition servers 200-1 and 200-2, respectively. That is, the partition server 200-1 is assigned a partition P1 from the master server 100, and the partition server 200-2 is assigned a partition P2 from the master server 100.

The master server 100 stores the path information of the files (P1.C1.LOG, P1.C2.LOG, P2.C1.LOG, P2.C2.LOG) in which the divided redo logs are recorded. 200-1,200-2) respectively. That is, the master server 100 delivers the partition information 200 to the partition server 200-1 together with the path information on the P1 related redo log files P1.C1.LOG and P1.C2.LOG. In addition, service partition P2 to partition server 200-2, and path information on the P2 related redo log files P2.C1.LOG and P2.C2.LOG.

Each partition server 200-1, 200-2,..., 200-n is responsible for servicing at least one partition and records the redo log of the update in one file (eg, redo log file). .

Each partition server 200-1, 200-2,..., 200-n receives a partition to be rebuilt and serviced from the master server 100, and a partitioned redo log that is the basis for rebuilding the partition. It receives the path information about the file.

Each partition server 200-1, 200-2,..., 200-n is a partition allocated from the master server 100 based on a redo log recorded in a partitioned redo log file corresponding to the received path information. Rebuild

When rebuilding a partition, each partition server 200-1,200-2, ..., 200-n responds to the partitioned redo log files with threads 200-1-1, ..., 200-1-n. , 200-2-1, ..., 200-2-n, ..., 200-n-1, ..., 200-nn, and the created threads (200-1-1 ,. (200-1-n, 200-2-1, ..., 200-2-n, ..., 200-n-1, ..., 200-nn) Data in parallel using the rerun logs.

For example, the selected partition server 200-1 may correspond to a thread (eg, 200-1-1,200-1-2) corresponding to the partitioned redo log files P1.C1.LOG and P1.C2.LOG. And partition the partition based on the redo logs recorded in the redo log files (P1.C1.LOG, P1.C2.LOG) partitioned from the created threads (200-1-1,200-1-2). Build, or recover data.

The selected partition server 200-2 generates threads (eg, 200-2-1,200-2-2) in response to the divided redo log files P2.C1.LOG and P2.C2.LOG. Rebuild the partition based on the redo logs recorded in the redo log files (P2.C1.LOG, P2.C2.LOG) split from the created threads (200-2-1,200-2-2), that is, data To recover.

Referring to FIG. 5, the ascending sorting of the redo log of the master server will be described. The redo log records created in the partition server 200-3 in which an error has occurred include the tables T1 and T2 and the row keys R1 and R2. , R3), columns C1, C2, log serial numbers (1, 2, 3, ..., 19, 20). You can see that the log records recorded in the redo log file before sorting are sorted in ascending order based on the log serial number.

The master server 100 first sorts the redo log records in ascending order, i.e. T1, T2, based on the table, and then in ascending order by row key, i.e., for R1, R2, R3, T2 for T1. Sort in R1, R2 order, then in ascending order by column: C1, C2 for T1, R1, C1, C2 for T1, R2, C2 for T1, R3, C1, C2 for T1, R1, Sort by C1, C2 for T2, R2.

The master server 100 classifies the redo logs sorted on the basis of partition configuration information for each partition, and classifies the redo logs classified for each partition by columns of the partition.

Referring to FIG. 6, the redo log partitioning of the master server is performed. The redo log records created by the partition server 200-3 in which an error has occurred include the tables T1, row keys R1, R2, R3, and R4, Columns (C1, C2), log serial numbers (1, 2, 3, ..., 19, 20). The redo logs are sorted in ascending order according to the procedure described with reference to FIG. 5, and the partitions in charge of services in the failed partition server 200-3 are P1 and P2 and the range of rows included in the preset partition configuration information. According to the information, partition P1 is greater than or equal to R1 and less than R3, and partition P2 is greater than or equal to R3 and less than R5.

The master server 100 classifies the redo logs sorted according to partitions P1 and P2 based on preset partition configuration information. The partition configuration information is reference information used for partitioning the partitions P1 and P2 and includes row range information of the partitions P1 and P2. That is, the partition configuration information includes row range information (eg, R1 <= P1 <R3, R3 <= P2 <R5) that tells which partition the log record written to the redo log file is the log record. do.

The master server 100 classifies the redo logs classified by partitions by columns C1 and C2 of partitions P1 and P2.

The master server 100 stores the rerun logs classified by the columns C1 and C2 of the partitions P1 and P2 as one file. For example, in FIG. 6, P1.C1.LOG is a log file that collects only log records for column C1 of partition P1.

7 is a flowchart illustrating a data recovery method in a cluster data management system according to the present invention.

Referring to FIG. 7, the master server 100 detects whether an error has occurred for each partition server 200-1, 200-2,..., 200-n (S700).

As a result of the detection, if an error occurs, the master server 100 generates a redo log generated by the failed partition server 200-3 based on preset reference information including a table, a row key, a column, and a log serial number. Sort in ascending order (S710).

The master server 100 divides the sorted redo logs by the columns C1 and C2 of the partitions P1 and P2 serviced by the partition server 200-3 (S720).

Log sorting and partitioning refer to the descriptions of FIGS. 5 and 6 described above.

The master server 100 selects the partition servers 200-1 and 200-2 to newly service the partitions P1 and P2 serviced by the failed partition server 200-3, and the failed partition server 200-. In step 3), the partitions P1 and P2 in charge of the service are allocated to the corresponding servers 200-1 and 200-2 (S730).

That is, the selected partition server 200-1 is assigned a partition P1 from the master server 100, and the selected partition server 200-2 is assigned a partition P2 from the master server 100.

The master server 100 is a redo log file (P1.C1.LOG, P1.C2.LOG, P2.C1.LOG, P2.C2.LOG) divided into partition servers 200-1 and 200-2 to which partitions are allocated. Pass in route information for.

The partition servers 200-1 and 200-2 are based on the redo logs recorded in the partitioned redo log files P1.C1.LOG, P1.C2.LOG, P2.C1.LOG, and P2.C2.LOG. The allocated partitions P1 and P2 are rebuilt (S740).

The partition servers 200-1 and 200-2 generate threads 200-1-1, 200-1-2, 200-2, 1-200-2-2 in response to the partitioned redo log files, and generate the created threads 200-1. Redo log files (P1.C1.LOG, P1.C2.LOG, P2.C1.LOG, P2.C2.LOG) partitioned via -1,200-1-2, 200-2-1,200-2-2) Recover data in parallel based on the redo log recorded in.

The partition servers 200-1 and 200-2 start the service for the partition from which data is recovered (S750).

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be defined not only by the scope of the following claims, but also by the equivalents of the claims.

1 shows a typical cluster data management system.

FIG. 2 is a diagram illustrating a data storage and service method of FIG. 1. FIG.

3 is a flowchart illustrating a data recovery method in a general cluster data management system.

4 is a block diagram illustrating an error recovery method in a cluster data management system according to the present invention;

FIG. 5 is a diagram illustrating rerun log sorting of a master server; FIG.

Fig. 6 is a diagram explaining replay log splitting of a master server.

7 is a flowchart illustrating a data recovery method in a cluster data management system according to the present invention.

DESCRIPTION OF THE REFERENCE NUMERALS OF THE DRAWINGS

100: master server 200-1: partition server

200-2: partition server 200-3: partition server

200-1-1-1: Thread 200-1-2: Thread

200-2-1: Thread 200-2-2: Three

Claims (18)

  1. In a data recovery method using parallel processing in a cluster data management system,
    Sorting the redo log created at the partition server where the error occurred;
    The redo logs sorted based on preset partition configuration information are classified by partitions that the partition server is in service, and then sorted again by the columns of each partition, and a file in which the redo logs classified by the columns is recorded. Dividing by column; And
    Selecting a partition server to serve partitions that are in service from the failed partition server, allocating a partition served by the failed partition server to the selected partition server, and path information of a file partitioned by the column; Reconstructing the partition assigned based on the log recorded in the file divided by the row corresponding to the path information in the selected partition server,
    Rebuilding the partition,
    Generating a thread in response to the partitioned file by the selected partition server; and
    Recovering data based on a log recorded in the file divided into rows by the generated threads;
    Data Recovery Using Parallel Processing in In-Cluster Data Management System.
  2. The method of claim 1, wherein the aligning step:
    And sorting the rerun log in ascending order based on predetermined reference information.
  3. The method of claim 2, wherein the predetermined reference information is
    A method of data recovery using parallel processing in a cluster data management system that includes a table, row key, column, and log serial number.
  4. delete
  5. The method of claim 1, wherein the partition configuration information,
    In the cluster data management system, the reference information used for partitioning includes row range information indicating whether each partition is greater than or equal to and less than or equal to which row among the row information included in the redo log. Data recovery method using parallel processing.
  6. delete
  7. delete
  8. delete
  9. The method of claim 1, wherein recovering the data comprises:
    A data recovery method using parallel processing in a cluster data management system in which at least one operator is allocated to each file divided by columns and processed in parallel.
  10. In a cluster data management system for recovering data using parallel processing,
    A partition server responsible for servicing at least one or more partitions, and recording a redo log according to the service of the partition;
    When the error occurs in the partition server, the redo logs arranged based on the partition configuration information preset by the partition are classified by the partitions, and then by the columns of the partitions, and the files containing the redo logs classified by the columns are divided by the columns. And selecting a partition server to rebuild the partition based on the partitioned redo log, allocating the partition, and including a master server to transfer path information about a file divided by columns.
    The partition server
    Reconstructing the partition allocated from the master server based on the log recorded in the divided file corresponding to the path information, generating a thread corresponding to the divided file, and partitioning through the generated thread. To recover data in parallel based on redo logs written to files
    Cluster data management system.
  11. The method of claim 10, wherein the master server,
    And sorting the redo log in ascending order based on predetermined reference information.
  12. The method of claim 11, wherein the preset reference information is:
    A cluster data management system that includes tables, row keys, columns, and log serial numbers.
  13. delete
  14. delete
  15. The method of claim 10, wherein the partition configuration information,
    And row range information indicating whether the partition is larger or equal to and smaller than or equal to which row among the row information included in the redo log as reference information used for the partitioning.
  16. delete
  17. delete
  18. delete
KR1020090024150A 2008-12-18 2009-03-20 Cluster data management system and method for data recovery using parallel processing in cluster data management system KR101259557B1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
KR20080129637 2008-12-18
KR1020080129637 2008-12-18

Publications (2)

Publication Number Publication Date
KR20100070968A KR20100070968A (en) 2010-06-28
KR101259557B1 true KR101259557B1 (en) 2013-04-30

Family

ID=42267529

Family Applications (1)

Application Number Title Priority Date Filing Date
KR1020090024150A KR101259557B1 (en) 2008-12-18 2009-03-20 Cluster data management system and method for data recovery using parallel processing in cluster data management system

Country Status (2)

Country Link
US (1) US20100161564A1 (en)
KR (1) KR101259557B1 (en)

Families Citing this family (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8566326B2 (en) * 2004-11-05 2013-10-22 Oracle International Corporation High-performance log-based processing
US7970820B1 (en) 2008-03-31 2011-06-28 Amazon Technologies, Inc. Locality based content distribution
US8601090B1 (en) 2008-03-31 2013-12-03 Amazon Technologies, Inc. Network resource identification
US7962597B2 (en) 2008-03-31 2011-06-14 Amazon Technologies, Inc. Request routing based on class
US8321568B2 (en) 2008-03-31 2012-11-27 Amazon Technologies, Inc. Content management
US8447831B1 (en) 2008-03-31 2013-05-21 Amazon Technologies, Inc. Incentive driven content delivery
US8028090B2 (en) 2008-11-17 2011-09-27 Amazon Technologies, Inc. Request routing utilizing client location information
US8688837B1 (en) 2009-03-27 2014-04-01 Amazon Technologies, Inc. Dynamically translating resource identifiers for request routing using popularity information
US8412823B1 (en) 2009-03-27 2013-04-02 Amazon Technologies, Inc. Managing tracking information entries in resource cache components
US8782236B1 (en) 2009-06-16 2014-07-15 Amazon Technologies, Inc. Managing resources using resource expiration data
US8433771B1 (en) * 2009-10-02 2013-04-30 Amazon Technologies, Inc. Distribution network with forward resource propagation
KR101313107B1 (en) * 2009-12-18 2013-09-30 한국전자통신연구원 Method and Apparatus for Managing Column Based Data
US9495338B1 (en) 2010-01-28 2016-11-15 Amazon Technologies, Inc. Content distribution network
US10467042B1 (en) 2011-04-27 2019-11-05 Amazon Technologies, Inc. Optimized deployment based upon customer locality
US10623408B1 (en) 2012-04-02 2020-04-14 Amazon Technologies, Inc. Context sensitive object management
US9154551B1 (en) 2012-06-11 2015-10-06 Amazon Technologies, Inc. Processing DNS queries to identify pre-processing information
US9323577B2 (en) 2012-09-20 2016-04-26 Amazon Technologies, Inc. Automated profiling of resource usage
US10205698B1 (en) 2012-12-19 2019-02-12 Amazon Technologies, Inc. Source-dependent address resolution
US9294391B1 (en) 2013-06-04 2016-03-22 Amazon Technologies, Inc. Managing network computing components utilizing request routing
US9819567B1 (en) 2015-03-30 2017-11-14 Amazon Technologies, Inc. Traffic surge management for points of presence
US9832141B1 (en) 2015-05-13 2017-11-28 Amazon Technologies, Inc. Routing based request correlation
JP2017068342A (en) * 2015-09-28 2017-04-06 富士通株式会社 Control program, control method, and information processor
US10348639B2 (en) 2015-12-18 2019-07-09 Amazon Technologies, Inc. Use of virtual endpoints to improve data transmission rates
US10075551B1 (en) 2016-06-06 2018-09-11 Amazon Technologies, Inc. Request management for hierarchical cache
US9992086B1 (en) 2016-08-23 2018-06-05 Amazon Technologies, Inc. External health checking of virtual private cloud network environments
US10033691B1 (en) 2016-08-24 2018-07-24 Amazon Technologies, Inc. Adaptive resolution of domain name requests in virtual private cloud network environments
US10469513B2 (en) 2016-10-05 2019-11-05 Amazon Technologies, Inc. Encrypted network addresses
US10372499B1 (en) 2016-12-27 2019-08-06 Amazon Technologies, Inc. Efficient region selection system for executing request-driven code
US10503613B1 (en) 2017-04-21 2019-12-10 Amazon Technologies, Inc. Efficient serving of resources during server unavailability
US10447648B2 (en) 2017-06-19 2019-10-15 Amazon Technologies, Inc. Assignment of a POP to a DNS resolver based on volume of communications over a link between client devices and the POP
CN107491314A (en) * 2017-08-30 2017-12-19 四川长虹电器股份有限公司 Processing method is write based on Read-Write Locks algorithm is accessible to HBASE real time datas
US10592578B1 (en) 2018-03-07 2020-03-17 Amazon Technologies, Inc. Predictive content push-enabled content delivery network

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5636360A (en) * 1991-03-28 1997-06-03 Texas Instruments Incorporated Method for preventing overwriting cache buffer transaction entries until corresponding log buffer entries have been copied to log partition of the disk
US7277900B1 (en) * 2000-09-29 2007-10-02 Oracle International Corporation Method and mechanism for rolling back a transaction on a row of data

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7177866B2 (en) * 2001-03-16 2007-02-13 Gravic, Inc. Asynchronous coordinated commit replication and dual write with replication transmission and locking of target database on updates only
US7305421B2 (en) * 2001-07-16 2007-12-04 Sap Ag Parallelized redo-only logging and recovery for highly available main memory database systems
JP3640187B2 (en) * 2002-07-29 2005-04-20 日本電気株式会社 Fault processing method for multiprocessor system, multiprocessor system and node
JP4561254B2 (en) * 2004-09-03 2010-10-13 セイコーエプソン株式会社 Device management system
US20080059492A1 (en) * 2006-08-31 2008-03-06 Tarin Stephen A Systems, methods, and storage structures for cached databases
US8880480B2 (en) * 2007-01-03 2014-11-04 Oracle International Corporation Method and apparatus for data rollback
US7711741B2 (en) * 2007-05-14 2010-05-04 Oracle International Corp. Desensitizing data in cloning

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5636360A (en) * 1991-03-28 1997-06-03 Texas Instruments Incorporated Method for preventing overwriting cache buffer transaction entries until corresponding log buffer entries have been copied to log partition of the disk
US7277900B1 (en) * 2000-09-29 2007-10-02 Oracle International Corporation Method and mechanism for rolling back a transaction on a row of data

Also Published As

Publication number Publication date
US20100161564A1 (en) 2010-06-24
KR20100070968A (en) 2010-06-28

Similar Documents

Publication Publication Date Title
JP6522812B2 (en) Fast Crash Recovery for Distributed Database Systems
JP6404907B2 (en) Efficient read replica
AU2017201918B2 (en) Prioritizing data reconstruction in distributed storage systems
JP6538780B2 (en) System-wide checkpoint avoidance for distributed database systems
JP6346937B2 (en) Database system with database engine and separate distributed storage service
JP2017102964A (en) Synchronous mirroring in non-volatile memory systems
EP2787446B1 (en) Distributed storage method, device and system
US20170132091A1 (en) Log-structured distributed storage using a single log sequence number space
US8862847B2 (en) Distributed storage method, apparatus, and system for reducing a data loss that may result from a single-point failure
CN101997918B (en) Method for allocating mass storage resources according to needs in heterogeneous SAN (Storage Area Network) environment
CN101676855B (en) Scalable secondary storage systems and methods
Borthakur et al. Apache hadoop goes realtime at facebook
US9116903B2 (en) Method and system for inserting data records into files
KR100490723B1 (en) Apparatus and method for file-level striping
US7587427B2 (en) Hierarchical storage system, and control method and program therefor
US8775763B2 (en) Redundant data assignment in a data storage system
JP5744244B2 (en) Storage system
US8843782B2 (en) Method and apparatus for reconstructing redundant array of inexpensive disks, and system
CN103116661B (en) A kind of data processing method of database
US9736243B2 (en) Multiple transaction logs in a distributed storage system
US9798471B2 (en) Performance of de-clustered disk array by disk grouping based on I/O statistics
KR100392382B1 (en) Method of The Logical Volume Manager supporting Dynamic Online resizing and Software RAID
US8417989B2 (en) Method and system for extra redundancy in a raid system
US7136964B2 (en) Disk array with spare logic drive created from spare physical drives
US6647460B2 (en) Storage device with I/O counter for partial data reallocation

Legal Events

Date Code Title Description
A201 Request for examination
E902 Notification of reason for refusal
E90F Notification of reason for final refusal
E701 Decision to grant or registration of patent right
GRNT Written decision to grant
FPAY Annual fee payment

Payment date: 20160226

Year of fee payment: 4

FPAY Annual fee payment

Payment date: 20170317

Year of fee payment: 5

FPAY Annual fee payment

Payment date: 20180411

Year of fee payment: 6

FPAY Annual fee payment

Payment date: 20190515

Year of fee payment: 7