CN112905535A - HBASE-based distributed object storage method - Google Patents

Abstract

The invention provides a distributed object storage system implementation method based on HBASE, which comprises the following steps: uploading: sending an upload request to the agent module; an authentication step: the agent module receives the request, authenticates and authenticates the request file and transmits the request file to the addressing module; a storage step: the addressing module stores the data of the request file in the storage system; an access step: and acquiring the information of the stored request file and generating a data stream return. The invention ensures high storage performance, effectively widens the adaptive scene of distributed storage, and can effectively avoid the problem of disk space waste.

Description

HBASE-based distributed object storage method

Technical Field

The invention relates to the field of data processing, in particular to a distributed object storage method based on HBASE.

Background

With the advance of informatization and intellectualization of industry, the data amount generated in industrial production and the value of production data are increasing day by day. For industrial scenarios, storing these data properly faces the following challenges:

1. challenge of data volume

The amount of data includes the total amount of data stored, as well as the throughput in processing such data. The traditional single-machine storage mode is limited by the hardware configuration bottleneck of the single machine. In the process of the increase of the storage data volume, the performance bottleneck can be met rapidly.

2. Challenge of reliability

In the process of informatization, the importance of data is increasing. In the intelligent stage, the data can directly influence the decision in the production link. Making the reliability requirements of the data an unavoidable challenge.

The existing Distributed storage (such as HDFS: Hadoop Distributed File System) can be well handled in terms of data volume and reliability. However, the application in industrial production still has the problems:

1. cannot cope with all file scenes

The existing file scenes responded by distributed storage are often limited, meanwhile, industrial production scenes are various, and data are more and more diversified. Exemplified by HDFS and HBASE.

In HDFS, files smaller than the configured block size are referred to as small files. Each small file occupies one block in the HDFS by itself, and a large number of small files results in a very large number of blocks at the same capacity. The metadata of each block is stored in the memory of the NameNode, which makes the memory occupation of the NameNode large. Accessing a large number of small files also results in significant performance degradation.

HBASE is a non-relational database with HDFS providing the underlying data storage. Due to design reasons, the storage scene of a large file cannot be dealt with.

2. The space occupied by the deleted file cannot be recovered in time

This problem occurs mainly in storage systems that use tree structures for storing data (e.g. HBASE). The data structure like the log structure merged tree can greatly improve the writing speed. However, in the storage system applying the structure, the deletion operation is mainly to mark the deleted data, and the data can be really deleted when the tree structures are combined. This results in the system not being able to release the storage space in time after the data is deleted.

Such storage systems can be wasteful of physical storage costs if data deletion is present in the primary scenario.

For the above reasons, the existing storage method cannot meet the application requirements, and we are forced to explore a distributed storage system more suitable for industrial scenarios.

For the problem that the existing distributed storage cannot cope with the diversification of the file size, the invention provides an implementation mode of a distributed storage system, which can keep the efficiency under the scenes of different data sizes and types and meet various industrial application requirements.

Patent document CN104750757B (application number: 201310753119.1) discloses an HBase-based data storage method and apparatus, including: receiving a query request message which is sent by an HBase and contains identification information of a Region in a Region Server with a fault, determining the identification information of a plurality of data storage copy data nodes corresponding to the identification information of the Region with the fault according to the corresponding relation between the identification information of the Region and the identification information of the data storage copy data nodes, sending a query response message to the HBase, enabling the HBase to reselect a new Region according to the query response message, and transferring data stored in the Region with the fault into the new Region.

Disclosure of Invention

Aiming at the defects in the prior art, the invention aims to provide a distributed object storage method based on HBASE.

The HBASE-based distributed object storage system implementation method provided by the invention comprises the following steps:

uploading: sending an upload request to the agent module;

an authentication step: the agent module receives the request, authenticates and authenticates the request file and transmits the request file to the addressing module;

a storage step: the addressing module stores the data of the request file in the storage system;

an access step: and acquiring the information of the stored request file and generating a data stream return.

Preferably, the storage structure adopted by the storage system is divided into two layers, namely a barrel and an object;

the request file is stored in a storage system in an object mode and is stored and accessed by an object name;

the object is: each object stores object metadata and object content data;

the barrel is as follows: is a container for objects, and isolates and distinguishes objects.

Preferably, the storage system stores the bucket table using a fixed table, and stores the object table using one fixed object table and a plurality of temporary object tables in combination;

the bucket table refers to information of a bucket;

the object table refers to object information;

the fixed object table: layering the file name and the content of the file;

the temporary object table: content data and metadata of a file or file block are stored.

Preferably, when the object is stored, the kind of the object stored in the temporary object table is determined by the life cycle of the object;

when the life cycle of the temporary object table is finished, deleting the whole table;

in the fixed object table, a combination of the name of the bucket, the delimiter, and the object name is stored as a row key, with a list of IDs of files or file blocks as a value;

in the temporary object table, the ID of a file or a file block is used as a row key, different column families are used for distinguishing and storing metadata and content data, and only the last block stores the metadata of the file in the same file block.

Preferably, the uploading refers to uploading through an access interface provided by the agent module;

the upload request includes: authentication information, file names, uploading target bucket names, uploading target object names, file contents, custom metadata and file life cycles;

the agent module: providing an access interface, requesting document authentication and authorization management;

the addressing module: and processing the writing and accessing processes of the file.

Preferably, the storing step comprises:

step M1: the method comprises the steps of segmenting a request file with the size exceeding an HBASE MOB threshold value to obtain a file block;

step M2: generating a unique ID for each file block;

step M3: storing the mapping relation between the position information and the ID stored in the file into a fixed object table;

step M4: calculating the name of a temporary object table corresponding to the life cycle of the file, and if the temporary object table does not exist, creating a new temporary object table;

step M5: storing the metadata of the file and the content data of the file block into a corresponding temporary object table;

step M6: and returning a storage result.

Preferably, the table name of the temporary object table is a combination of the corresponding fixed object table name and a time stamp;

the time stamp represents: the corresponding temporary object table is deleted by the whole table at the time point corresponding to the time stamp.

Preferably, the timestamp is calculated by:

if the time stamp of the file request uploading is ct, the requested file is marked to be deleted when the time point dt is ct + t, wherein,

dt represents the point in time at which the requested file is marked for deletion;

t represents the life cycle length of the uploaded request file;

for N belongs to N, the minimum value of N T meeting N T > dt is obtained, and the minimum value is the time stamp in the temporary object table;

wherein the content of the first and second substances,

n represents a positive integer;

n denotes a positive integer set.

Preferably, the accessing step comprises:

step N1: acquiring a block storage position and a block ID list from a fixed object table;

step N2: sequentially inquiring block data corresponding to the block ID from the temporary object table;

step N3: encapsulating the data block obtained by query into a data stream;

step N4: the data stream is returned.

Preferably, the block data includes: metadata and content data.

Compared with the prior art, the invention has the following beneficial effects:

1. the invention ensures high storage performance and effectively widens the adaptive scene of distributed storage.

2. The invention can be applied to any xInsight item.

3. The invention can effectively avoid the problem of disk space waste.

Drawings

Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:

fig. 1 is a schematic structural diagram of a storage system according to the present invention.

Fig. 2 is a schematic view of an uploading process provided by the present invention.

Fig. 3 is a schematic view of the access (download) process provided by the present invention.

FIG. 4 is a schematic diagram of an interface S3 in the storage of the Browser accessing xIns light object provided by the present invention.

FIG. 5 is a schematic diagram of a management page of a web page based on a Restful interface in an xInsight object store provided by the present invention.

FIG. 6 is a schematic diagram of an authorization management page of a web page based on a Restful interface in an xInsight object store provided by the present invention.

Detailed Description

The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.

The HBASE-based distributed object storage system implementation method provided by the invention comprises the following steps:

uploading: sending an upload request to the agent module;

an authentication step: the agent module receives the request, authenticates and authenticates the request file and transmits the request file to the addressing module;

a storage step: the addressing module stores the data of the request file in the storage system;

an access step: and acquiring the information of the stored request file and generating a data stream return.

Specifically, the storage structure adopted by the storage system is divided into two layers, namely a bucket and an object;

the request file is stored in a storage system in an object mode and is stored and accessed by an object name;

the object is: each object stores object metadata and object content data;

the barrel is as follows: is a container for objects, and isolates and distinguishes objects.

Specifically, the storage system stores a bucket table using a fixed table, stores an object table using one fixed object table and a plurality of temporary object tables in combination;

the bucket table refers to information of a bucket;

the object table refers to object information;

the fixed object table: layering the file name and the content of the file;

the temporary object table: content data and metadata of a file or file block are stored.

Specifically, when the object is stored, the type of the object stored in the temporary object table is determined by the life cycle of the object;

when the life cycle of the temporary object table is finished, deleting the whole table;

in the fixed object table, a combination of the name of the bucket, the delimiter, and the object name is stored as a row key, with a list of IDs of files or file blocks as a value;

in the temporary object table, the ID of a file or a file block is used as a row key, different column families are used for distinguishing and storing metadata and content data, and only the last block stores the metadata of the file in the same file block.

Specifically, the uploading refers to uploading through an access interface provided by the agent module;

the upload request includes: authentication information, file names, uploading target bucket names, uploading target object names, file contents, custom metadata and file life cycles;

the agent module: providing an access interface, requesting document authentication and authorization management;

the addressing module: and processing the writing and accessing processes of the file.

Specifically, the storing step includes:

step M1: the method comprises the steps of segmenting a request file with the size exceeding an HBASE MOB threshold value to obtain a file block;

step M2: generating a unique ID for each file block;

step M3: storing the mapping relation between the position information and the ID stored in the file into a fixed object table;

step M4: calculating the name of a temporary object table corresponding to the life cycle of the file, and if the temporary object table does not exist, creating a new temporary object table;

step M5: storing the metadata of the file and the content data of the file block into a corresponding temporary object table;

step M6: and returning a storage result.

Specifically, the table name of the temporary object table is a combination of a corresponding fixed object table name and a timestamp;

the time stamp represents: the corresponding temporary object table is deleted by the whole table at the time point corresponding to the time stamp.

Specifically, the timestamp is obtained by calculation in the following way:

if the time stamp of the file request uploading is ct, the requested file is marked to be deleted when the time point dt is ct + t, wherein,

dt represents the point in time at which the requested file is marked for deletion;

t represents the life cycle length of the uploaded request file;

for N belongs to N, the minimum value of N T meeting N T > dt is obtained, and the minimum value is the time stamp in the temporary object table;

wherein the content of the first and second substances,

n represents a positive integer;

n denotes a positive integer set.

Specifically, the accessing step includes:

step N1: acquiring a block storage position and a block ID list from a fixed object table;

step N2: sequentially inquiring block data corresponding to the block ID from the temporary object table;

step N3: encapsulating the data block obtained by query into a data stream;

step N4: the data stream is returned.

Specifically, the block data includes: metadata and content data.

The present invention will be described more specifically below with reference to preferred examples.

Preferred example 1:

object storage service in Xin Bao self-research big data platform software xInsight

The implementation mode is as follows:

and 1, in a proxy module realized by storing the xInsight object, an S3 interface and a Restful interface are realized. And providing a webpage end management page based on the Restful interface. A legend for accessing the S3 interface using the S3 client tool is shown in fig. 4. The management page of the web page side is illustrated in fig. 5.

The xInsight object store enables user authentication and authorization management in the proxy module. Fine-grained privilege control can be performed on the operation of uploading/accessing/managing objects by a user. An illustration of the rights control in the administration page of the web site is shown in fig. 6.

Supporting multiple tenants in the xInsight, and realizing the management of users by the tenants. The addressing module is implemented according to the principle of isolation among tenants. I.e., each tenant has a separate bucket table and object table.

Preferred example 2:

HBASE introduces HBASE MOB characteristics in HBASE-11339. By separating the IO paths of the reference file and the MOB object, the intermediate object can be better stored. The invention realizes distributed object storage for various file sizes by utilizing HBASE and HBASE MOB characteristics newly added to HBASE. And the problem that the space cannot be released in time after the HBASE file is deleted is solved.

The invention mainly comprises two parts, namely an agent module and an addressing module. The agent module is responsible for providing interaction with the user, such as the S3 interface, Restful interface, and user authentication and authorization management. The addressing module is used for processing the writing and accessing process of the file. The basic structure is shown in figure 1. The scheme mainly explains the addressing module part and the main scene flow.

1. Memory structure

The storage structure adopted is divided into two layers, namely a bucket and an object.

The file is stored in the storage system in an object mode, and is stored and accessed by an object name. Each object will store object metadata and object content data.

A bucket is a container of objects. The purpose is to isolate and differentiate objects according to the usage scenario.

In this scheme, the HBASE uses a fixed Table (Table) to store the information of the bucket (bucket Table), and uses a fixed Table (fixed object Table) and a plurality of temporary tables (temporary object tables) to store the object information (object Table) in combination.

In the object table, the role of the fixed object table is to layer the file name and the content of the file. Because the name of the file name is determined by the user, if the name is directly used as a row key of HBASE, hot spot problems can be caused; the temporary object table stores content data and metadata of files or file blocks. When the object is stored, the life cycle of the object determines the temporary object table into which the object is stored. And when the life cycle of the temporary table is finished, the whole table is deleted.

In the fixed object table, a combination of the name of the bucket, the delimiter, and the object name is stored as a Row Key (Row Key) with a list of IDs of files or file blocks as a value.

In the temporary object table, the storage metadata and the content data are distinguished by different column families with the ID of a file or a file block as a row key. Wherein, in the blocks of the same file, only the last block stores the metadata of the file.

Attribute character strings meeting preset conditions in the data to be processed can be obtained according to requirements; performing first deformation processing on the attribute character string to obtain a first deformed character string; performing second deformation processing on the attribute character string to obtain a second deformed character string; and processing the first deformed character string and the second deformed character string again to obtain a final used result character string, and storing the result character string as the RowKey of the data stored in the HBase cluster.

2. File uploading process

When the user uploads the file, the file is uploaded through the access interface provided by the agent module. The upload request includes: user authentication information, file name, uploading target bucket name, uploading target object name, file content, custom metadata and file life cycle.

And after receiving the request, the agent module authenticates and authenticates the user information. After the file is passed, the file related information is transferred to an addressing module for processing.

The addressing module processing steps are as follows, and are illustrated in fig. 2:

a. segmenting files with the size exceeding the HBASE MOB threshold value;

b. generating a unique ID for each file block (the file which is not segmented is regarded as a block);

c. storing the mapping relation between the position information and the ID stored in the file into a fixed object table;

d. calculating a temporary object table name corresponding to the life cycle of the file, and if the temporary object table does not exist, creating the temporary object table;

e. storing the metadata of the file and the content data of the file block into a corresponding temporary object table;

f. and returning a storage result. The returned storage result is whether the storage result of the user is successful or not and the failure reason in the failure scene. Returning by a user access mode (for example, the user performs a storage action through a Web management page, and after the user clicks and triggers the storage, the storage result can be returned to the user as a feedback result of the clicking action)

The table name of the temporary object table is a combination of the corresponding fixed object table name and a time stamp. The meaning of the timestamp is to indicate that the table is to be deleted by the whole table at the time point corresponding to the timestamp. When the file is uploaded, the calculation mode of the time stamp in the corresponding temporary object table name is as follows:

the system can be configured with a table management period of T and an uploaded file life cycle length of T. If the time stamp uploaded by the file is ct, the file is marked to be deleted when dt is ct + t. And for N ∈ N ∈, calculating the minimum value of N ∈ T satisfying N ∈ T > dt, namely obtaining the timestamp in the temporary object table.

3. File access scenarios

When the user accesses the file, the user accesses the file through the access interface of the agent module. The access request includes: user authentication information, and a path of a file object to be accessed.

In the uploading process, the agent module authenticates and authenticates the user information after receiving the request. The addressing module then retrieves the stored file information and generates a data stream return.

The specific process is as follows, and the figure is figure 3:

a. acquiring a block storage position and a block ID list from a fixed object table according to the path information of the file object;

b. sequentially inquiring block data (metadata and content data) corresponding to the block ID from the temporary object table;

c. packaging the file data obtained by query into a data stream;

d. and returning the data stream to the user.

4. Optimization of space release in a scenario where a file is deleted for a lifecycle

HBASE only performs marker deletion when files are deleted due to the use of a log-structured merge tree. And the data marked for deletion is deleted from the tree only when the tree structures are merged. However, the merging is a very expensive operation, and the process may occupy a lot of system resources, which is an operation that cannot be performed frequently.

This may result in the deleted file remaining on the disk for a long time, resulting in wasted disk space. In an industrial production scene, a lot of data are time-efficient and need to be cleaned regularly.

In the scheme, the file data is stored in the temporary object table, the data expired in the same time window is stored in the same temporary object table, and the whole table is deleted after the data expires. Since the deletion of the entire table is equivalent to the direct deletion of the entire tree structure, the space can be released without a merge action. The problem of disk space waste can be effectively avoided.

Those skilled in the art will appreciate that, in addition to implementing the systems, apparatus, and various modules thereof provided by the present invention in purely computer readable program code, the same procedures can be implemented entirely by logically programming method steps such that the systems, apparatus, and various modules thereof are provided in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system, the device and the modules thereof provided by the present invention can be considered as a hardware component, and the modules included in the system, the device and the modules thereof for implementing various programs can also be considered as structures in the hardware component; modules for performing various functions may also be considered to be both software programs for performing the methods and structures within hardware components.

The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.

Claims (10)

1. A distributed object storage system implementation method based on HBASE is characterized by comprising the following steps:

uploading: sending an upload request to the agent module;

an authentication step: the agent module receives the request, authenticates and authenticates the request file and transmits the request file to the addressing module;

a storage step: the addressing module stores the data of the request file in the storage system;

an access step: and acquiring the information of the stored request file and generating a data stream return.

2. The HBASE-based distributed object storage system implementing method according to claim 1, wherein the storage structure adopted by the storage system is divided into two layers, namely a bucket and an object;

the request file is stored in a storage system in an object mode and is stored and accessed by an object name;

the object is: each object stores object metadata and object content data;

the barrel is as follows: is a container for objects, and isolates and distinguishes objects.

3. The HBASE-based distributed object storage system implementing method of claim 2, wherein the storage system uses fixed tables to store bucket tables, one fixed object table and multiple temporary object tables in combination with storage object tables;

the bucket table refers to information of a bucket;

the object table refers to object information;

the fixed object table: layering the file name and the content of the file;

the temporary object table: content data and metadata of a file or file block are stored.

4. The HBASE-based distributed object storage system implementing method of claim 3, wherein the object, when stored, determines the kind of object stored in the temporary object table from the object's lifecycle;

when the life cycle of the temporary object table is finished, deleting the whole table;

in the fixed object table, a combination of the name of the bucket, the delimiter, and the object name is stored as a row key, with a list of IDs of files or file blocks as a value;

in the temporary object table, the ID of a file or a file block is used as a row key, different column families are used for distinguishing and storing metadata and content data, and only the last block stores the metadata of the file in the same file block.

5. The HBASE-based distributed object storage system implementing method of claim 1, wherein the uploading is via an access interface provided by a proxy module;

the upload request includes: authentication information, file names, uploading target bucket names, uploading target object names, file contents, custom metadata and file life cycles;

the agent module: providing an access interface, requesting document authentication and authorization management;

the addressing module: and processing the writing and accessing processes of the file.

6. The HBASE-based distributed object storage system implementing method according to claim 1, wherein said storing step comprises:

step M1: the method comprises the steps of segmenting a request file with the size exceeding an HBASE MOB threshold value to obtain a file block;

step M2: generating a unique ID for each file block;

step M3: storing the mapping relation between the position information and the ID stored in the file into a fixed object table;

step M4: calculating the name of a temporary object table corresponding to the life cycle of the file, and if the temporary object table does not exist, creating a new temporary object table;

step M5: storing the metadata of the file and the content data of the file block into a corresponding temporary object table;

step M6: and returning a storage result.

7. The HBASE-based distributed object storage system implementing method of claim 6, wherein the table name of the temporary object table is a combination of a corresponding fixed object table name and a timestamp;

the time stamp represents: the corresponding temporary object table is deleted by the whole table at the time point corresponding to the time stamp.

8. The HBASE-based distributed object storage system implementing method of claim 7, wherein the time stamp is calculated by:

if the time stamp of the file request uploading is ct, the requested file is marked to be deleted when the time point dt is ct + t, wherein,

dt represents the point in time at which the requested file is marked for deletion;

t represents the life cycle length of the uploaded request file;

for N belongs to N, the minimum value of N T meeting N T > dt is obtained, and the minimum value is the time stamp in the temporary object table;

wherein the content of the first and second substances,

n represents a positive integer;

n denotes a positive integer set.

9. The HBASE-based distributed object storage system implementing method according to claim 1, wherein said accessing step comprises:

step N1: acquiring a block storage position and a block ID list from a fixed object table;

step N2: sequentially inquiring block data corresponding to the block ID from the temporary object table;

step N3: encapsulating the data block obtained by query into a data stream;

step N4: the data stream is returned.

10. The HBASE-based distributed object storage system implementing method of claim 9, wherein the block data comprises: metadata and content data.

Images