CN113672171A

CN113672171A - Distributed object storage method, device and system and metadata server

Info

Publication number: CN113672171A
Application number: CN202110845526.XA
Authority: CN
Inventors: 王志豪; 罗心; 江文龙; 周明伟
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2021-07-26
Filing date: 2021-07-26
Publication date: 2021-11-19

Abstract

The application relates to a distributed object storage method, a device, a system and a metadata server, wherein the method comprises the following steps: when an object application request initiated by a client is received, acquiring a writable area of an SMR disk in a distributed storage system; the object application request comprises the size of a file to be written, and the SMR disk is divided into a plurality of writable areas; determining writing information according to the object request and the residual storage space of the writable area; the writing information comprises the length of each object obtained by segmenting the file to be written; and returning the writing information to the client so that the client writes the file to be written into the corresponding storage node according to the writing information. By the method and the device, the aggregate storage management of the Zone space is realized under the condition that the length of the object is variable, the Zone space can be used to the maximum extent, and the waste of the storage space is reduced.

Description

Distributed object storage method, device and system and metadata server

Technical Field

The present application relates to the field of distributed object storage technologies, and in particular, to a distributed object storage method, apparatus, system, and metadata server.

Background

With the continuous development of Storage technology, Object-based Storage (OBS) is produced in order to meet the Storage requirements of the network cluster system which are changing. The object storage is a novel storage structure, a distributed object storage file system can be established based on the storage structure, and the distributed object storage file system has cross-platform data sharing, fine-grained application data management, high performance and good scalability.

In a distributed file system, when a user writes a file, the file is split into fixed size objects and a fixed space size is allocated. Generally, the object length is generally defined to be relatively large, and as shown in fig. 1, writing a file of 400MB requires 4 objects to be stored, and the object is often not full. At this point there would be no significant problem if the objects were stored on a conventional CMR disk. However, when the SMR disk (split Magnetic Recording) is used for storage, since the SMR disk manages a minimum granularity as a storage area Zone, the Zone has a fixed size of 256MB, only additional writing is allowed inside the Zone, and deletion of data requires erasing of all the zones in a block, and erasing of partial data and random writing are not allowed. If writing is performed based on the existing manner, each object writes one Zone, and the Zone is not fully written, which results in a very large waste of space.

For the serious problem of space waste, in the prior art, a Zone is multiplexed during file writing, an unfilled Zone is fully written, and then the Zone is switched to the next Zone, at this time, a file storage schematic diagram is shown in fig. 2, and at this time, for writing a file of 400MB, space waste is greatly improved, but 112MB space waste still exists. Still further, if the unused 112MB space in the Zone needs to be multiplexed, the size of the object in the current distributed storage system is fixed, and at this time, only 112MB space cannot accommodate a complete object. However, if one object is stored in two different zones, it causes a lot of trouble in the following lifecycle management, data recovery, and metadata management.

Disclosure of Invention

The embodiment of the application provides a distributed object storage method, a device, a system and a metadata server, so as to at least solve the problem of space waste during object storage in the related technology.

In a first aspect, an embodiment of the present application provides a distributed object storage method, including:

when an object application request initiated by a client is received, acquiring a writable area of an SMR disk in a distributed storage system; the object application request comprises the size of a file to be written, and the SMR disk is divided into a plurality of writable areas;

determining writing information according to the object request and the residual storage space of the writable area; the writing information comprises the length of each object obtained by segmenting the file to be written;

and returning the writing information to the client so that the client writes the file to be written into the corresponding storage node according to the writing information.

In some embodiments, determining write information according to the object request and the remaining storage space of the writable area comprises:

determining a plurality of first objects according to the object application request and a preset object length;

acquiring the residual storage space of the writable area, and determining the adjustment information of the object length according to the residual storage space;

adjusting the size and length of the first object according to the adjustment information to obtain a second object;

and pre-allocating a capacity space according to the length of the second object, and allocating a writable area according to the pre-allocated capacity space to obtain writing information corresponding to the file to be written.

In some embodiments, obtaining the remaining storage space of the writable area, and determining the adjustment information of the object length according to the remaining storage space includes:

acquiring the residual storage space of the writable area;

when the remaining storage space of the writable area is smaller than the object length, determining the adjustment information of the object length as the reduction of the object length; wherein the reduced object length is not less than the data size within the object.

In some embodiments, when the remaining storage space of the writable area is smaller than the object length, the adjustment information for determining the object length is:

reducing the object length to a data size within the object.

In some embodiments, pre-allocating a volume space according to the length of the second object, and allocating a writable area according to the pre-allocated volume space to obtain writing information corresponding to a file to be written includes:

acquiring at least one writable area;

acquiring a corresponding pre-allocated capacity space according to the length of each second object;

writing the corresponding second objects into the residual storage space of the writable area in sequence according to the pre-allocated capacity space;

determining writing information corresponding to a file to be written according to the written writable area; index information of the writable area and a length of each second object in the writable area.

In some embodiments, returning the write information to the client, so that after the client writes the file to be written into the corresponding storage node according to the write information, the method further includes:

acquiring the residual storage space of the writable area uploaded by the storage node; and after the file to be written is written into the storage node, the storage node calculates the residual storage space based on the pre-allocated capacity space of at least one second object in the writable area.

acquiring the length of each second object actually written into the writable area;

and when the length of the second object actually written into the writable area is smaller than the corresponding pre-allocated capacity space, correcting the pre-allocated capacity space to be the length of the second object actually written into the writable area.

In some embodiments, the writing information further includes the number of objects in the writable area, and the method further includes:

persistently storing the write information;

and when the number of the objects in the writable area is zero, deleting the corresponding writable area.

In a second aspect, an embodiment of the present application provides a distributed object storage apparatus, including:

the writable area acquisition unit is used for acquiring the writable area of the SMR disk in the distributed storage system when receiving an object application request initiated by a client; the object application request comprises the size of a file to be written, and the SMR disk is divided into a plurality of writable areas;

a write information determination unit configured to determine write information according to the object request and the remaining storage space of the writable area; the writing information comprises the length of each object obtained by segmenting the file to be written;

and the storage unit is used for returning the writing information to the client so that the client writes the file to be written into the corresponding storage node according to the writing information.

In a third aspect, an embodiment of the present application provides a metadata server, including a memory and a processor, where the memory stores a computer program, and the processor implements the distributed object storage method according to the first aspect when executing the computer program.

In a fourth aspect, an embodiment of the present application provides a distributed object storage system, including: the system comprises a client, at least one storage node and a metadata server; the metadata server is in communication connection with each storage node and the client;

the client is used for initiating an object application request to the metadata server and writing the file to be written into the corresponding storage node according to the writing information returned by the metadata server;

the storage node is used for storing the file to be written in the client;

the metadata server is configured to implement the distributed object storage method according to the first aspect.

In some of these embodiments, the storage node is further configured to:

and calculating the residual storage space of the corresponding writable area based on the pre-allocated capacity space of the second object in each writable area, and uploading the residual storage space of the writable area to the metadata server.

Compared with the related art, the distributed object storage method provided by the embodiment of the application determines the write-in information according to the object application request and the remaining storage space of the writable area, wherein the write-in information comprises the length of each object obtained by segmenting the file to be written, so that the object length is determined according to the write-in request and the remaining storage space of the writable area, the object length is variable, and the space management of the object is facilitated. And returning the writing information to the client so that the client writes the file to be written into the corresponding storage node according to the writing information. The method and the device realize the aggregate storage management of the Zone space under the condition that the object length is variable, can maximally use the Zone space, and reduce the waste of the storage space.

The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the application.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:

FIG. 1 is a schematic diagram of a file storage in the prior art;

FIG. 2 is a schematic diagram of another file storage in the prior art;

FIG. 3 is a schematic diagram of an application scenario of the distributed object storage method in one embodiment of the present application;

FIG. 4 is a schematic flow chart diagram illustrating a distributed object storage method according to an embodiment of the present application;

FIG. 5 is a schematic diagram of the storage of documents 1 and 2 in one embodiment of the present application;

FIG. 6 is a flow chart illustrating the determination of written information according to one embodiment of the present application;

FIG. 7 is a block diagram of a distributed object store in one embodiment of the present application;

FIG. 8 is a block diagram of a metadata server in one embodiment of the present application;

FIG. 9 is a block diagram of a distributed object storage system in one embodiment of the present application.

Description of the drawings: 101. a metadata server; 102. a storage node; 103. a client; 701. a writable area acquisition unit; 702. a write information determination unit; 703. a storage unit; 80. a bus; 81. a processor; 82. a memory; 83. a communication interface.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments provided in the present application without any inventive step are within the scope of protection of the present application.

It is obvious that the drawings in the following description are only examples or embodiments of the present application, and that it is also possible for a person skilled in the art to apply the present application to other similar contexts on the basis of these drawings without inventive effort. Moreover, it should be appreciated that in the development of any such actual implementation, as in any engineering or design project, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which may vary from one implementation to another.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the specification. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of ordinary skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments without conflict.

Unless defined otherwise, technical or scientific terms referred to herein shall have the ordinary meaning as understood by those of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar words throughout this application are not to be construed as limiting in number, and may refer to the singular or the plural. The present application is directed to the use of the terms "including," "comprising," "having," and any variations thereof, which are intended to cover non-exclusive inclusions; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to the listed steps or elements, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. Reference to "connected," "coupled," and the like in this application is not intended to be limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as referred to herein means two or more. "and/or" describes an association relationship of associated objects, meaning that three relationships may exist, for example, "A and/or B" may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. Reference herein to the terms "first," "second," "third," and the like, are merely to distinguish similar objects and do not denote a particular ordering for the objects.

Distributed File System (DFS) means that physical storage resources managed by a File System are not necessarily directly connected to a local node, but are connected to a node through a computer network; or a complete hierarchical file system formed by combining several different logical disk partitions or volume labels. The distributed file system can effectively solve the storage and management problems of data, a certain file system fixed at a certain place is expanded to any multiple places/multiple file systems, and a file system network is formed by a plurality of nodes. Each node may be distributed at different locations, with communication and data transfer between nodes over the network.

SMR (Shingled Magnetic Recording) disks are high capacity disks that employ a new type of Magnetic storage technology, which overlaps data tracks on the disks partially, like tiles on a roof, and is called Shingled Magnetic Recording technology. The technology has very small change in the manufacturing process, but can greatly improve the storage density of the magnetic disk and effectively reduce the storage cost of the magnetic disk per unit capacity.

The distributed object storage method provided by the present application may be applied to a distributed file system based on an SMR disk, and as shown in fig. 3, an application scene diagram of the distributed object storage method provided by the embodiment of the present application is shown. As shown in fig. 3, the metadata server is in communication connection with each storage node and the client, the client sends an object application request to the metadata server, and the metadata server allocates storage information and returns the storage information to the client according to the object application request and the writable area of each communication node. And the client requests and completes the write-in operation to the corresponding storage node according to the storage information.

The present embodiment provides a distributed object storage method, which may be executed by a metadata server in a distributed object storage system, and fig. 4 is a flowchart of the distributed object storage method according to an embodiment of the present application, and as shown in fig. 4, the flowchart includes the following steps:

step S101, when receiving an object application request initiated by a client, acquiring a writable area of an SMR disk in a distributed storage system; the object application request comprises the size of a file to be written, and the SMR disk is divided into a plurality of writable areas.

An object is a basic unit of data storage, consisting of meta-information, user data and a file name. In a storage node of a distributed storage system, an SMR disk may be divided into a plurality of writable areas zones, space management of the SMR disk is granular with the size of one Zone (e.g., 256MB), and the Zone is used to store an object.

In this embodiment, when a client requests to store a file, the file to be written needs to be split into a plurality of objects, and then the objects are stored in the writable area Zone of the SMR disk in the distributed object storage system. Specifically, the client generates an object application request according to the file to be written, and then initiates the object application request to the metadata server. When the metadata server receives an object application request initiated by a client, a writable area Zone of an SMR disk in the distributed storage system is acquired, and subsequent storage space allocation is carried out according to the writable area Zone. Alternatively, the client may be a desktop computer, a cell phone, a PDA, or the like.

In this embodiment, the object application request at least includes the size of the file to be written. Of course, in other embodiments, the object application request may further include a device identifier or other custom information of the client, and the application is not particularly limited.

Step S102, determining writing information according to the object request and the residual storage space of the writable area; the writing information comprises the length of each object obtained by segmenting the file to be written.

In the related art, the length of each object obtained by splitting the file to be written is fixed, so that the object is often not fully written or the writable area Zone is not fully written, and under the background that data grows exponentially, each storage node has great space waste.

In this embodiment, in order to improve the space utilization of the writable area Zone, the write information may be determined according to the object request and the remaining storage space of the writable area Zone. Specifically, the size of the file to be written may be determined according to the object application request, the length of each object obtained by segmenting the file to be written is adaptively configured according to the size of the file to be written and the remaining storage space of the writable area, and the object with a variable length is subjected to space management, so as to achieve maximum use of a Zone space. It can be understood that the writing information includes lengths of objects obtained by splitting the file to be written, and also includes index information necessary for writing and storing the file to be written, where the index information includes a node ID, a Zone ID, an object ID, and the like, and the present application is not limited specifically herein.

For example, as shown in fig. 5, a file 1 of 400MB is written, and the lengths of the objects obtained by segmenting the file to be written are adaptively configured according to the size of the file to be written and the remaining storage space of the writable area, so that 4 objects are required to be stored, including an object 1(128MB), an object 2(128MB), an object 3(128MB), and an object 4(16 MB); writing a file 2 of 368MB, and adaptively configuring and segmenting the length of each object obtained by the file to be written according to the size of the file to be written and the remaining storage space of the writable area, wherein 3 objects are required to be stored, including an object 1(112MB), an object 2(128MB) and an object 3(128 MB). During writing, the file 1 object 1 and the file 1 object 2 may be written into the first Zone, the file 1 object 3, the file 1 object 4, and the file 2 object 1 are written into the second Zone, and the file 2 object 2 and the file 2 object 3 are written into the third Zone, so that the lengths of the objects obtained by segmenting the file to be written are adaptively configured according to the size of the file to be written and the remaining storage space of the writable area, and thus, aggregated storage of the objects is realized.

Step S103, returning the writing information to the client so that the client writes the file to be written into the corresponding storage node according to the writing information.

Specifically, in this embodiment, after determining the write information corresponding to the file to be written, the metadata server returns the write information to the client. And after receiving the write-in information, the client sends a write-in request to the corresponding storage node according to the write-in information. And after the storage node receives the writing request, writing the file to be written into the corresponding writable area.

In summary, according to the distributed object storage method provided by the embodiment of the present application, the write-in information is determined according to the object application request and the remaining storage space of the writable area, where the write-in information includes the length of each object obtained by segmenting the file to be written, so that the object length is determined according to the write-in request and the remaining storage space of the writable area, the object length is variable, and the space management of the object is facilitated. And returning the writing information to the client so that the client writes the file to be written into the corresponding storage node according to the writing information. The method and the device realize the aggregate storage management of the Zone space under the condition that the object length is variable, can maximally use the Zone space, and reduce the space waste.

The embodiments of the present application are described and illustrated below by means of preferred embodiments.

As shown in fig. 6, on the basis of the foregoing embodiments, in some embodiments, the determining write information according to the object request and the remaining storage space of the writable area includes:

step S1021, determining a plurality of first objects according to the object application request and the preset object length.

Specifically, in this embodiment, a to-be-written file is obtained according to the object application request, and then the to-be-written file is segmented according to a preset object length to obtain a plurality of first objects. The first objects are obtained by dividing according to preset object lengths, and the sizes of the first objects are the same. The method is limited by the limitations of a memory, a CPU, a disk, and the like of the device where the metadata service is located, and in order to reduce the amount of metadata as much as possible and increase the data storage amount, the lengths of the preset object length objects are generally defined to be large, such as 64MB, 128MB, 256MB, and the like, and the preset object lengths may be configured by a user, which is not specifically limited herein.

Step S1022, acquiring the remaining storage space of the writable area, and determining the adjustment information of the object length according to the remaining storage space.

Step S1023, adjusting the size and length of the first object according to the adjustment information to obtain a second object.

In this embodiment, the adjustment information of the object length may be dynamically configured according to the remaining storage space of the writable area, and the size and length of the first object are adjusted according to the adjustment information, so that the writable area Zone is multiplexed in the writing process, and the second object with the adjusted length is stored in one writable area Zone in a gathering manner, thereby increasing the data storage capacity of the writable area.

Specifically, the obtaining of the remaining storage space of the writable area, and determining the adjustment information of the object length according to the remaining storage space includes: acquiring the residual storage space of the writable area; and when the remaining storage space of the writable area is smaller than the object length, determining the adjustment information of the object length as the reduction of the object length. In some embodiments, the adjustment information of the object length is that the reduced object length is smaller than the length of the first object and larger than the data size in the object. In other embodiments, the adjustment information of the object length is: reducing the object length to a data size within the object. The adjustment information of the object length may be configured by user, and the application is not limited specifically herein.

Step S1024, pre-allocating a capacity space according to the length of the second object, and allocating a writable area according to the pre-allocated capacity space to obtain writing information corresponding to the file to be written.

In some specific embodiments, at least one writable area is first obtained, a corresponding pre-allocation capacity space is obtained according to the length of each second object, and pre-allocation of a storage space is performed according to the second object after length adjustment. Wherein the pre-allocated volume space is not less than a length corresponding to the second object. For example, in some embodiments, the size of the pre-allocated volume space may be consistent with the length of the corresponding second object.

Secondly, distributing a writable area according to the pre-allocated capacity space of each second object, and sequentially writing the corresponding second objects into the residual storage space of the writable area according to the pre-allocated capacity space; in some embodiments, the writable area zones corresponding to the second object may be randomly allocated and sequentially written, and when one writable area Zone is full or cannot be continuously written, another writable area Zone is continuously written.

And finally, determining the writing information corresponding to the file to be written according to the written writable area. The writing information is index information of the writable area and the length of each second object in the writable area. The index information may be, for example, a writable area ID and a second object ID within the writable area.

On the basis of the foregoing embodiments, in some embodiments, after returning the write information to the client, so that the client writes the file to be written into the corresponding storage node according to the write information, the method further includes:

In this embodiment, after the file to be written is written, the remaining storage space corresponding to the writable area is calculated and updated according to the pre-allocated capacity space of the second object. And if the updated residual storage space is not zero, reporting the writable area to a metadata server for caching. Therefore, when the client initiates the object application request again, the written information such as the object length can be determined according to the writable area.

In this embodiment, there may be a deviation between the pre-allocated capacity space according to the length of the second object and the length of the second object actually written into the writable area, so that after the file to be written is written into the corresponding storage node, a challenge needs to be posed on space management. Specifically, after the file to be written is written, the length of the second object actually written into the writable area is compared with the corresponding pre-allocated capacity space, and when the length of the second object actually written into the writable area is smaller than the corresponding pre-allocated capacity space, the pre-allocated capacity space needs to be corrected to the length of the second object actually written into the writable area, so that accurate capacity calculation is ensured.

On the basis of the foregoing embodiments, in some of the embodiments, the write information further includes the number of objects in the writable area, and then the method further includes: and persistently storing the write-in information, and deleting the corresponding writable area when the number of the objects in the writable area is zero.

In this embodiment, after the file to be written is written, persistent storage is performed on the written information, so as to ensure reliability of the distributed object storage system. Specifically, the write information further includes the number of objects in the writable area, and when the number of objects in the writable area is zero, the corresponding writable area is deleted. Specifically, when an object is deleted, if an object cannot monopolize a Zone, space release cannot be performed, and space release can be performed only after all objects are deleted, so that the corresponding writable area is deleted.

It should be noted that the steps illustrated in the above-described flow diagrams or in the flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order different than here.

The present embodiment further provides a distributed object storage apparatus, which is used to implement the foregoing embodiments and preferred embodiments, and the description of the apparatus is omitted here. As used hereinafter, the terms "module," "unit," "subunit," and the like may implement a combination of software and/or hardware for a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.

Fig. 7 is a block diagram of a distributed object storage apparatus according to an embodiment of the present application, and as shown in fig. 7, the apparatus includes: a writable area acquisition unit 701, a write information determination unit 702, and a storage unit 703.

A writable area obtaining unit 701, configured to obtain a writable area of an SMR disk in a distributed storage system when an object application request initiated by a client is received; the object application request comprises the size of a file to be written, and the SMR disk is divided into a plurality of writable areas;

a write information determining unit 702, configured to determine write information according to the object request and the remaining storage space of the writable area; the writing information comprises the length of each object obtained by segmenting the file to be written;

the storage unit 703 is configured to return the write information to the client, so that the client writes the file to be written into the corresponding storage node according to the write information.

In some of these embodiments, the write information determination unit 702 includes: the device comprises a first object determination module, an adjustment information determination module, a second object determination module and a writable area allocation module.

The first object determining module is used for determining a plurality of first objects according to the object application request and the preset object length;

an adjustment information determining module, configured to obtain a remaining storage space of the writable area, and determine adjustment information of an object length according to the remaining storage space;

the second object determining module is used for adjusting the size and the length of the first object according to the adjusting information to obtain a second object;

and the writable area distribution module is used for pre-distributing the capacity space according to the length of the second object and distributing the writable area according to the pre-distributed capacity space to obtain the writing information corresponding to the file to be written.

In some embodiments, the adjustment information determining module comprises: the device comprises a residual storage space determining module and a first adjusting information obtaining module.

A remaining storage space determining module, configured to obtain a remaining storage space of the writable area;

a first adjustment information obtaining module, configured to determine, when a remaining storage space of the writable area is smaller than an object length, that adjustment information of the object length is to reduce the object length; wherein the reduced object length is not less than the data size within the object.

In some embodiments, the adjustment information determining module further comprises: and a second adjustment information acquisition module.

And the second adjustment information acquisition module is used for reducing the length of the object to the size of the data in the object.

In some of these embodiments, the writable area allocation module includes: the device comprises a writable area acquisition module, a capacity space pre-allocation module, a writing module and a writing information determination module.

A writable area obtaining module, configured to obtain at least one writable area;

the volume space pre-allocation module is used for acquiring a corresponding pre-allocated volume space according to the length of each second object;

the writing module is used for sequentially writing the corresponding second objects into the residual storage space of the writable area according to the pre-allocated capacity space;

the write-in information determining module is used for determining write-in information corresponding to a file to be written according to the written writable area; index information of the writable area and a length of each second object in the writable area.

In some of these embodiments, the distributed object store further comprises: and a remaining storage space acquisition unit.

A remaining storage space acquiring unit, configured to acquire a remaining storage space of the writable area uploaded by the storage node; and after the file to be written is written into the storage node, the storage node calculates the residual storage space based on the pre-allocated capacity space of at least one second object in the writable area.

In some of these embodiments, the distributed object store further comprises: a length acquisition unit and a spatial correction unit.

A length acquisition unit configured to acquire a length of each of the second objects actually written in the writable area;

and the space correction unit is used for correcting the pre-allocated capacity space to be the length of the second object which is actually written into the writable area when the length of the second object which is actually written into the writable area is smaller than the corresponding pre-allocated capacity space.

In some embodiments, the writing information further includes the number of objects in the writable area, and the distributed object storage apparatus further includes: a persistence unit and a deletion unit.

A persistence unit configured to persistently store the write information;

and the deleting unit is used for deleting the corresponding writable area when the number of the objects in the writable area is zero.

The above modules may be functional modules or program modules, and may be implemented by software or hardware. For a module implemented by hardware, the modules may be located in the same processor; or the modules can be respectively positioned in different processors in any combination.

In addition, the distributed object storage method described in conjunction with fig. 1 in the embodiment of the present application may be implemented by a metadata server. Fig. 8 is a hardware configuration diagram of a metadata server according to an embodiment of the present application.

The metadata server may include a processor 81 and a memory 82 storing computer program instructions.

Specifically, the processor 81 may include a Central Processing Unit (CPU), or A Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.

Memory 82 may include, among other things, mass storage for data or instructions. By way of example, and not limitation, memory 82 may include a Hard Disk Drive (Hard Disk Drive, abbreviated to HDD), a floppy Disk Drive, a Solid State Drive (SSD), flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 82 may include removable or non-removable (or fixed) media, where appropriate. The memory 82 may be internal or external to the data processing apparatus, where appropriate. In a particular embodiment, the memory 82 is a Non-Volatile (Non-Volatile) memory. In particular embodiments, Memory 82 includes Read-Only Memory (ROM) and Random Access Memory (RAM). The ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Electrically Erasable PROM (EEPROM), Electrically rewritable ROM (EAROM), or FLASH Memory (FLASH), or a combination of two or more of these, where appropriate. The RAM may be a Static Random-Access Memory (SRAM) or a Dynamic Random-Access Memory (DRAM), where the DRAM may be a Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), an Extended data output Dynamic Random-Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), and the like.

The memory 82 may be used to store or cache various data files for processing and/or communication use, as well as possible computer program instructions executed by the processor 81.

The processor 81 implements any of the distributed object storage methods in the above embodiments by reading and executing computer program instructions stored in the memory 82.

In some of these embodiments, the metadata server may also include a communication interface 83 and a bus 80. As shown in fig. 8, the processor 81, the memory 82, and the communication interface 83 are connected via the bus 80 to complete communication therebetween.

The communication interface 83 is used for implementing communication between modules, devices, units and/or equipment in the embodiment of the present application. The communication interface 83 may also enable communication with other components such as: the data communication is carried out among external equipment, image/data acquisition equipment, a database, external storage, an image/data processing workstation and the like.

Bus 80 includes hardware, software, or both to couple the components of the metadata server to each other. Bus 80 includes, but is not limited to, at least one of the following: data Bus (Data Bus), Address Bus (Address Bus), Control Bus (Control Bus), Expansion Bus (Expansion Bus), and Local Bus (Local Bus). By way of example, and not limitation, Bus 80 may include an Accelerated Graphics Port (AGP) or other Graphics Bus, an Enhanced Industry Standard Architecture (EISA) Bus, a Front-Side Bus (FSB), a Hyper Transport (HT) Interconnect, an ISA (ISA) Bus, an InfiniBand (InfiniBand) Interconnect, a Low Pin Count (LPC) Bus, a memory Bus, a microchannel Architecture (MCA) Bus, a PCI (Peripheral Component Interconnect) Bus, a PCI-Express (PCI-X) Bus, a Serial Advanced Technology Attachment (SATA) Bus, a Video Electronics Bus (audio Electronics Association), abbreviated VLB) bus or other suitable bus or a combination of two or more of these. Bus 80 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.

The metadata server may execute the distributed object storage method in the embodiment of the present application based on the obtained program instruction, thereby implementing the distributed object storage method described in conjunction with fig. 4.

As shown in fig. 9, this embodiment further provides a distributed object storage system, including: a client 103, at least one storage node 102, and a metadata server 101; the metadata server 101 is in communication connection with each storage node 102 and the client 103.

In this embodiment, the client 103 is configured to initiate an object application request to the metadata server 101, and write the file to be written into the corresponding storage node 102 according to the write information returned by the metadata server 101. The storage node 102 is configured to store a file to be written in the client. Specifically, the client sends a write request to the metadata server 101, and when the storage node 102 receives the write request, writes the file to be written into the corresponding writable area. And after the writing is finished, returning the writing success information. The metadata server 101 is configured to execute the distributed object storage method, which is not described herein again.

In some embodiments, the storage node 102 is further configured to: and calculating the residual storage space of the corresponding writable area based on the pre-allocated capacity space of the second object in each writable area, and uploading the residual storage space of the writable area to the metadata server 101. In this embodiment, after receiving a write request sent by the client 103, the storage node 102 searches for a corresponding Zone according to the write request, and checks whether the remaining size of the Zone meets the object length, if the check is passed, the file to be stored is additionally written in the Zone, and the metadata is recorded, and the remaining storage space of the writable area is uploaded to the metadata server 101.

In addition, in combination with the distributed object storage method in the foregoing embodiments, the embodiments of the present application may provide a computer-readable storage medium to implement. The computer readable storage medium having stored thereon computer program instructions; the computer program instructions, when executed by a processor, implement any of the distributed object storage methods of the embodiments described above.

The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.

The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A distributed object storage method, comprising:

2. The distributed object storage method of claim 1, wherein determining write information based on the object request and the remaining storage space of the writable area comprises:

3. The distributed object storage method according to claim 2, wherein obtaining the remaining storage space of the writable area and determining the adjustment information of the object length according to the remaining storage space comprises:

acquiring the residual storage space of the writable area;

4. The distributed object storage method according to claim 3, wherein when the remaining storage space of the writable area is smaller than the object length, the adjustment information for determining the object length is:

reducing the object length to a data size within the object.

5. The distributed object storage method according to claim 2, wherein pre-allocating a volume space according to the length of the second object, and allocating a writable area according to the pre-allocated volume space, and obtaining write information corresponding to a file to be written comprises:

acquiring at least one writable area;

6. The distributed object storage method according to claim 1, wherein after returning the write information to the client to enable the client to write the file to be written into the corresponding storage node according to the write information, the method further includes:

7. The distributed object storage method according to claim 2, wherein after returning the write information to the client to enable the client to write the file to be written into the corresponding storage node according to the write information, the method further includes:

8. The distributed object storage method of claim 1, wherein the write information further includes the number of objects in the writable area, the method further comprising:

persistently storing the write information;

9. A distributed object storage apparatus, comprising:

10. A metadata server comprising a memory and a processor, wherein the memory has stored therein a computer program, and the processor is arranged to run the computer program to perform the distributed object storage method of any of claims 1 to 8.

11. A distributed object storage system, comprising: the system comprises a client, at least one storage node and a metadata server; the metadata server is in communication connection with each storage node and the client;

the storage node is used for storing the file to be written in the client;

the metadata server is configured to perform the distributed object storage method of any one of claims 1 to 8.

12. The distributed object storage system of claim 11, wherein the storage node is further configured to: