WO2017028721A1

WO2017028721A1 - Data update method and device in distributed file system

Info

Publication number: WO2017028721A1
Application number: PCT/CN2016/094322
Authority: WO
Inventors: 段兵
Original assignee: 阿里巴巴集团控股有限公司
Priority date: 2015-08-19
Filing date: 2016-08-10
Publication date: 2017-02-23
Also published as: CN106469172B; CN106469172A

Abstract

A data update method and device in a distributed file system. The method comprises: acquiring update data, writing the update data to a tail portion of a current data node, and updating corresponding index information in the data node (S402); and writing the update data to a tail portion of a check node, and updating corresponding index information in the check node (S404). The present invention achieves the effects of high performance, less computational resource occupation and data consistency for a data update process in a distributed file system.

Description

Data update method and device for distributed file system

The present application claims priority to Chinese Patent Application Serial No. No. No. No. No. No. No. No. No. No. No. No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No No

Technical field

The present application relates to the field of computer technologies, and in particular, to a data update method and apparatus for a distributed file system.

Background technique

With the continuous development of the Internet, data on the Internet has grown explosively. The cost is also getting higher and higher. Many distributed file systems encode data by the Erasure Code algorithm to achieve cost reduction. There are many problems in applying the Erasure Code update algorithm in the distributed file system, which cannot be applied in the production environment, but the business implementation is inseparable from the file update operation, so how to solve the problem caused by the Erasure Code update algorithm is also coming. The more forced.

Assuming that the parameters encoded by the Erasure Code are: the data node is M and the checkpoint is N, the disadvantages of the prior art algorithm are:

(1) low performance;

a read and write disk times: the disk reads M-1 data, write M + N data to the disk;

b Large amount of network data transmission: receiving M-1 data from the network and sending M-1+N data to the network;

(2) The computing resources are wasted seriously, and the CPU needs to calculate M data;

(3) The data cannot be recovered when part of the failure occurs during the update process; for example, when the data is written to the disk after the calculation is completed, such problems may occur if there is a failure.

In summary, the application of the Erasure Code update algorithm in the distributed file system for data update results in high cost, low performance, and difficulty in ensuring consistency. Therefore, it is necessary to propose improved technical means to solve the above problems.

Summary of the invention

The main purpose of the present application is to provide a data update method and apparatus for a distributed file system, which overcomes the problems of high cost, low performance, and difficulty in ensuring consistency of data distribution in the distributed file system existing in the prior art.

A data update method for a distributed file system according to an embodiment of the present application is provided. Include a plurality of data nodes and at least one check node, the method comprising: obtaining update data, writing the update data to a tail of a current data node, and updating corresponding index information in the data node; The update data is written to the tail of the check node, and the corresponding index information is updated in the check node.

The updating the corresponding index information in the data node includes: updating an end position occupied by a storage space of the data node; finding an index of the original data corresponding to the update data, and indexing the index Modified to the index of the update data.

The method further includes: sending the update data, the identifier information of the original data corresponding to the update data, and the identifier information of the current data node to the check node, where the identifier information of the data node includes: Disk ID, IP address, and port information.

The updating the corresponding index information in the check node includes: updating an end position of the occupied storage space in the check node.

The method further includes: querying, according to the identifier information of the data node, index information corresponding to the data node; searching, according to the identifier information of the original data corresponding to the update data, in the index information of the data node To the index of the original data, the index is modified to an index of the updated data.

According to an embodiment of the present application, a data update apparatus of a distributed file system is further provided, where the distributed file system includes a plurality of data nodes and at least one check node, and the apparatus includes:

An obtaining module, configured to obtain update data; a data node update module, configured to write the update data to a tail of a current data node, and update corresponding index information in the data node; and verify a node update module, where The update data is written to the tail of the check node, and the corresponding index information is updated in the check node.

The data node update module is further configured to: update an end position occupied by the storage space of the data node; find an index of the original data corresponding to the update data, and modify the index to the update data. index of.

The method further includes: a data sending module, configured to send the update data, the identification information of the original data corresponding to the update data, and the identifier information of the current data node to the check node, where the data The identification information of the node includes: disk identifier, IP address, and port information.

The check node update module is further configured to update an end position of the occupied storage space in the check node.

The check node update module is further configured to: query, according to the identifier information of the data node, index information corresponding to the data node; and according to the identifier information of the original data corresponding to the update data, An index of the original data is found in the index information of the data node, and the index is modified into an index of the update data.

According to the above technical solution of the present application, by writing the update data to the data node and checking the tail of the node, the defect of applying the Erasure Code update algorithm for data update is solved by the disk space, in the data update process of the distributed file system. Effectively achieves high performance, no use of computing resources, and consistent data.

DRAWINGS

The drawings described herein are intended to provide a further understanding of the present application, and are intended to be a part of this application. In the drawing:

1 is an architectural diagram of a distributed file system in accordance with an embodiment of the present application;

2 is a schematic diagram of storage management of a data node according to an embodiment of the present application;

3 is a schematic diagram of storage management of a check node according to an embodiment of the present application;

4 is a flowchart of a data update method of a distributed file system according to an embodiment of the present application;

FIG. 5 is a schematic diagram of data update according to an embodiment of the present application; FIG.

6 is a structural block diagram of a data update apparatus of a distributed file system according to an embodiment of the present application;

FIG. 7 is a structural block diagram of a data updating apparatus of a distributed file system according to another embodiment of the present application.

detailed description

The technical solutions of the present application will be clearly and completely described in the following with reference to the specific embodiments of the present application and the corresponding drawings. It is apparent that the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope are the scope of the present application.

A data update method for a distributed file system is provided according to an embodiment of the present application.

The present application is built on a distributed file system. Referring to FIG. 1, a distributed file system is composed of at least one master node (or a control node), at least one client, and multiple storage nodes. The devices communicate with each other through a network. Each node is an independent physical machine. The main control node is mainly used to locate storage nodes, and each storage node is responsible for managing one disk.

The storage node includes two types: a storage original data storage node and a storage verification data storage node:

The storage raw data storage node, referred to as a data node, is used to store the original data. The raw data is divided into different data blocks according to a predetermined size (for example, 1M), and each small block has a unique number (ID) with respect to the current data node.

The functions of the data node mainly include: (1) receiving and forwarding from the master node, other data nodes and/or guest Network data of the client; (2) Management of data on the disk and index information.

In an actual application, the index information of the data node specifically includes: a disk ID, an encoding start position and an encoding end position, an ending position of the occupied storage space (incremented sequentially), and an index of all the data blocks. Referring to FIG. 2, each data block 21 corresponds to an index 22, and the index of the data block includes: a data block identifier (ID), a data block start position, and a data block end position. In one embodiment of the present application, the location of the data file (data block) in the data node can be quickly located by the index information of the data block.

Referring to FIG. 3, a checksum data storage node, which is simply referred to as a check node, stores check data 31 generated by the Erasure Code algorithm and update data 32 generated by the update file, wherein the update data is written to the end of the check node. .

The check node and the data node not only store different data types, but also store the type of index information. As shown in FIG. 3, the check node usually records the encoding start position, the end position, and the end position occupied by the storage space (incrementally incremented) of the check node at the initial position 33 of the index data, and then after the start position 33. Location 44 stores index information for all other data nodes. As shown in FIG. 3, the check node records the disk identifier, IP address, port information (disk information) of each data node, and the index of the check data block corresponding to the data block of the data node. The index of the check data block may include: a check data block identifier (ID), a check data block start position, and a check data block end position. In practical applications, the position of the check data block in the check node can be quickly located by checking the index of the data block.

Referring to FIG. 4, FIG. 4 illustrates a flow chart of a data update method of a distributed file system according to an embodiment of the present application.

Step S402, acquiring update data, writing the update data to a tail of the current data node, and updating corresponding index information in the data node;

After the update data is written to the end of the data node, the storage space of the data node changes, so the end position of the occupied storage space of the data node needs to be updated; and after the data is updated, the index of the original data needs to be updated. Position, specifically: updating the start position of the update data to N, the end position update to N plus update data length, and the end position occupied by the storage space is updated to N plus update data length; wherein N is the pre-update storage space The end position occupied.

Then, the update data, the identification information of the original data corresponding to the update data, and the identifier information of the current data node are sent to the check node, where the identifier information of the data node includes: a disk identifier, IP address and port information.

Step S404, writing the update data to the tail of the check node, and updating the pair in the check node Index information should be.

After receiving the update data, the check node appends the data to its own tail, and then needs to update the index information of the check node, specifically: querying the index information corresponding to the data node according to the identifier information of the data node; And searching, according to the identifier information of the original data corresponding to the update data, an index of the original data in the index information of the data node, and modifying the index into an index of the update data.

First, the end position of the storage space is updated to N plus the new data length; then the index information related to the data node is queried according to the identification information of the data node, and the start position of the update data is updated to N and the end The location is updated to N plus the update data length.

An example of data update of a distributed file system is described in detail below in conjunction with FIG.

In an embodiment of the present application, the distributed file system may be a TFS (Taobao File System), and the TFS system manages storage of data files in units of data blocks, each of which has a data block. Globally unique identification information (ID). When storing small files, multiple small files can be stored in the same data block, and the TFS system supports the update operation of the files. When the TFS system stores a file, it establishes index information according to the file. The index information is not encoded in the Erasure Code and is directly copied to the data node (check node) where the parity block is located.

In the encoding process, only the real data of each data block is Erasure Code encoded, and the index information of each data block is not encoded. The index information of each data block is copied to each check block (check node) and stored in the index file of the check block.

In order to briefly describe the update data flow, the distributed file system is set to include three data nodes and one check node (only two data nodes are shown in FIG. 5 for simplicity), the encoding start position is 0, the end position is M, and the storage is performed. The end position where the space is occupied is N. It is assumed that the data node 1 and the data block numbered 2 are updated, and the update data (new data) is acquired in advance. Referring to FIG. 5, the update procedure of the data block 1 of the data node 1 (ie, the original data) is as follows:

(1) The data node 1 appends new data to its own tail and updates the index information related to the data block No. 2. In data node 1, the index of data block 2 is index 2, before the update, index 2 points to data block 2 (shown in the dotted line in the figure); after the update, index 2 needs to point to the update data, that is, The start position of index 2 is updated to N, the end position is updated to N+ new data length (shown as the solid line segment in the figure), and the end position of the storage space is updated to N+ new data length;

(2) The data node 1 transmits new data, identification information of the original data, and related information of the data node 1 (disk ID, IP address, port number PORT) to the check node 1 through the network;

(3) In the check node 1, before the update, the index 2 points to the data block 2 (that is, the dotted line segment in the figure); after the update, the check node 1 first adds the new data to itself after receiving the new data. At the same time, the end position of the updated storage space is N+ new data length; the index information related to the data node 1 is queried by the disk ID, the IP address, and the port number, and the index information of the data block 2 is updated, and the index 2 points to the update. Data (shown in the solid line in the figure).

According to the embodiment of the present application, the following technical effects can be achieved:

(1) high performance;

The number of read and write disks only accounts for the original scheme (2/(M-1+M+N)=2/(3-1+3+1)=2/6=33.33%;

Network traffic only accounts for the original scheme (2/(M-1+M+N)=2/(3-1+3+1)=2/6=33.33%;

(2) does not occupy computing resources, only need to add data at the end of the disk;

(3) When the partial failure of the update process occurs, the data that the user sees is consistent.

According to an embodiment of the present application, there is also provided a data update apparatus of a distributed file system, where the distributed file system includes a plurality of data nodes and at least one check node.

FIG. 6 is a structural block diagram of a data update apparatus of a distributed file system according to an embodiment of the present application. As shown in FIG. 6, the apparatus includes:

The obtaining module 610 is configured to obtain update data.

a data node update module 620, configured to write the update data to a tail of a current data node, and update corresponding index information in the data node;

Further, the data node update module 620 updates an end position occupied by the storage space of the data node; finds an index of the original data corresponding to the update data, and modifies the index into an index of the update data. . Specifically, the start position of the update data is updated to N, the end position update is N plus the update data length, and the end position occupied by the storage space is updated to N plus update data length; wherein N is the pre-update occupied storage. The end position of the space.

The check node update module 630 is configured to write the update data to the tail of the check node, and update the corresponding index information in the check node.

Further, the check node update module 630 updates the end position of the occupied storage space in the check node; and queries the index information corresponding to the data node according to the identification information of the data node; Updating the identification information of the original data corresponding to the data, searching an index of the original data in the index information of the data node, and modifying the index to an index of the updated data. Specifically, the end position of the storage space is updated to N plus a new data length; and an index related to the data node is queried according to the identification information of the data node. Information; the start position of the update data is updated to N, and the end position is updated to N plus update data length.

Referring to FIG. 7, FIG. 7 is a structural block diagram of a data updating apparatus of a distributed file system according to another embodiment of the present application, where the apparatus includes: an obtaining module 710, a data node updating module 720, a check node updating module 730, Data sending module 740.

The obtaining module 710, the data node updating module 720, and the check node updating module 730 are similar to the obtaining module 610, the data node updating module 620, and the check node updating module 630, respectively, and are not described herein.

As shown in FIG. 7, the data sending module 740 is configured to send the update data, the identification information of the original data corresponding to the update data, and the identifier information of the current data node to the check node, where The identification information of the data node includes: a disk identifier, an IP address, and port information.

The operational steps of the method of the present invention correspond to the structural features of the device, and can be referred to each other without further elaboration.

In summary, according to the above technical solution of the present application, by writing update data to the data node and checking the tail of the node, the disk space is used to solve the defect that the application of the Erasure Code update algorithm for data update exists in the distributed file system. The data update process effectively achieves high performance, no use of computing resources, and consistent data.

Those skilled in the art will appreciate that embodiments of the present application can be provided as a method, system, or computer program product. Thus, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment in combination of software and hardware. Moreover, the application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory. Memory is an example of a computer readable medium.

Computer readable media includes both permanent and non-persistent, removable and non-removable media. Information storage can be implemented by any method or technology. The information can be computer readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory. (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disk read only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, Magnetic box magnetic A tape, magnetic tape storage or other magnetic storage device or any other non-transportable medium can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary storage of computer readable media, such as modulated data signals and carrier waves.

It is also to be understood that the terms "comprises" or "comprising" or "comprising" or any other variations are intended to encompass a non-exclusive inclusion, such that a process, method, article, Other elements not explicitly listed, or elements that are inherent to such a process, method, commodity, or equipment. An element defined by the phrase "comprising a ..." does not exclude the presence of additional equivalent elements in the process, method, item, or device including the element.

The above description is only an embodiment of the present application and is not intended to limit the application. Various changes and modifications can be made to the present application by those skilled in the art. Any modifications, equivalents, improvements, etc. made within the spirit and scope of the present application are intended to be included within the scope of the appended claims.

Claims

A data update method for a distributed file system, the distributed file system comprising a plurality of data nodes and at least one check node, wherein the method comprises:

Obtaining update data, writing the update data to a tail of a current data node, and updating corresponding index information in the data node;

The update data is written to the tail of the check node, and the corresponding index information is updated in the check node.
The method according to claim 1, wherein the updating the corresponding index information in the data node comprises:

Updating an end position occupied by the storage space of the data node;

An index of the original data corresponding to the update data is found, and the index is modified to an index of the update data.
The method of claim 1 further comprising:

Sending the update data, the identifier information of the original data corresponding to the update data, and the identifier information of the current data node to the check node, where the identifier information of the data node includes: a disk identifier, an IP address Port information.
The method according to claim 3, wherein the updating the corresponding index information in the check node comprises:

Updating the end position of the occupied storage space in the check node.
The method of claim 4, further comprising:

Querying, according to the identification information of the data node, index information corresponding to the data node;

And searching, according to the identifier information of the original data corresponding to the update data, an index of the original data in the index information of the data node, and modifying the index into an index of the update data.
A data update device for a distributed file system, the distributed file system comprising a plurality of data nodes and at least one check node, wherein the device comprises:

An acquisition module for obtaining update data;

a data node update module, configured to write the update data to a tail of a current data node, and update corresponding index information in the data node;

And a check node update module, configured to write the update data to a tail of the check node, and update corresponding index information in the check node.
The device according to claim 6, wherein the data node update module is further configured to: update an end position occupied by a storage space of the data node; and find original data corresponding to the update data. An index that modifies the index to an index of the updated data.
The device according to claim 6, further comprising:

a data sending module, configured to send, to the check node, the update data, the identifier information of the original data corresponding to the update data, and the identifier information of the current data node, where the identifier information of the data node includes : Disk ID, IP address, and port information.
The apparatus according to claim 8, wherein the check node update module is further configured to update an end position of the occupied storage space in the check node.
The apparatus according to claim 6, wherein the check node update module is further configured to: query, according to the identification information of the data node, index information corresponding to the data node; according to the update data Corresponding identification information of the original data, an index of the original data is found in the index information of the data node, and the index is modified into an index of the update data.