CN111610936B

CN111610936B - Object storage platform, object aggregation method and device and server

Info

Publication number: CN111610936B
Application number: CN202010450286.9A
Authority: CN
Inventors: 郭军; 李金阳
Original assignee: Guangzhou Baiguoyuan Information Technology Co Ltd
Current assignee: Bigo Technology Singapore Pte Ltd
Priority date: 2020-05-25
Filing date: 2020-05-25
Publication date: 2023-04-14
Anticipated expiration: 2040-05-25
Also published as: WO2021238408A1; CN111610936A

Abstract

The embodiment of the invention discloses an object storage platform, an object aggregation method, an object aggregation device and a server. Wherein, this platform includes: the system comprises an object storage gateway, a first storage pool and a second storage pool, wherein the first storage pool stores a first class object which is not aggregated to a second class object and an aggregation mapping relation between the first class object which is aggregated to the second class object and the aggregated second class object, and the second storage pool stores the second class object which is aggregated by the first class object; the object storage gateway periodically aggregates the first class objects which are not aggregated to the second class objects in the first storage pool into the second class objects in the second storage pool, and replaces the aggregated first class objects in the first storage pool with the aggregated mapping relation between the first class objects and the aggregated second class objects. The technical scheme provided by the embodiment of the invention optimizes the storage structure of the first class object and improves the reading performance of the first class object which is not aggregated to the second class object.

Description

Object storage platform, object aggregation method and device and server

Technical Field

The embodiment of the invention relates to the technical field of data processing, in particular to an object storage platform, an object aggregation method, an object aggregation device and a server.

Background

With the gradual increase of multimedia resources (such as pictures, audios and videos and the like), the current open-source Ceph file system has corresponding storage and operation requirements for file objects with different sizes, the Ceph file system allocates a minimum space unit for data storage, and even if the data volume of a small object is smaller than the minimum space unit, the small object can completely occupy the minimum space unit, so that the storage space is greatly wasted; meanwhile, when the Ceph file system is subjected to capacity expansion or hardware failure, the file objects stored on the Ceph file system are usually transferred by taking the supported minimum operation granularity as a unit, at this time, for small objects with data volume lower than the minimum operation granularity, the Ceph file system may have a risk of data loss when performing conversion operation, and the data read-write efficiency of the Ceph file system is greatly increased by the transfer of a large number of small objects.

At present, when a Ceph file system merges a plurality of small objects into one large object for storage, the Ceph file system additionally stores a mapping relationship between a custom object name of each small object and a merging position in the large object by using a third-party storage system for the small objects with the data volume lower than the minimum operation granularity.

Disclosure of Invention

The embodiment of the invention provides an object storage platform, an object aggregation method, an object aggregation device and a server, which are used for optimizing the storage structure of a first class of objects and reducing the complexity of object storage.

In a first aspect, an embodiment of the present invention provides an object storage platform, where the platform includes: the system comprises an object storage gateway, a first storage pool and a second storage pool, wherein the first storage pool is a storage space constructed by adopting a solid-state disk technology on an internal storage space, and stores a first class object which is not aggregated to a second class object and an aggregation mapping relation between the first class object which is aggregated to the second class object and the aggregated second class object, and the second storage pool is a storage space constructed by adopting a conventional disk technology on the internal storage space and stores the aggregated second class object of the first class object; wherein the content of the first and second substances,

and the object storage gateway periodically aggregates the first class objects which are not aggregated to the second class objects in the first storage pool into the second class objects in the second storage pool, and replaces the aggregated first class objects in the first storage pool with the aggregated mapping relation between the first class objects and the aggregated second class objects.

In a second aspect, an embodiment of the present invention provides an object aggregation method, which is applied to the object storage platform provided in the first aspect, and the method includes:

periodically finding out first class objects which are not aggregated to second class objects in a first storage pool;

aggregating the first class objects which are not aggregated to the second class objects into the second class objects in the second storage pool, and replacing the aggregated first class objects in the first storage pool with the aggregated mapping relation between the first class objects and the aggregated second class objects;

the first storage pool is a storage space constructed by adopting a solid-state disk technology on an internal storage space, and stores a first class object which is not aggregated to a second class object, and an aggregation mapping relation between the first class object which is aggregated to the second class object and the aggregated second class object, and the second storage pool is a storage space constructed by adopting a conventional disk technology on the internal storage space and stores a second class object after the aggregation of the first class object.

In a third aspect, an embodiment of the present invention provides an object aggregation apparatus, which is disposed in the object storage platform provided in the first aspect, and includes:

the object searching module is used for regularly searching a first class of objects which are not aggregated to a second class of objects in the first storage pool;

the object aggregation module is used for aggregating a first class of objects which are not aggregated to a second class of objects into the second class of objects in the second storage pool, and replacing the aggregated first class of objects in the first storage pool with an aggregation mapping relation between the first class of objects and the aggregated second class of objects;

the first storage pool is a storage space constructed by adopting a solid-state disk technology on an internal storage space, and stores a first class object which is not aggregated to a second class object and an aggregation mapping relation between the first class object which is aggregated to the second class object and the aggregated second class object, and the second storage pool is a storage space constructed by adopting a conventional disk technology on the internal storage space and stores the aggregated second class object of the first class object.

In a fourth aspect, an embodiment of the present invention provides a server, where the server includes:

one or more processors;

storage means for storing one or more programs;

when executed by the one or more processors, cause the one or more processors to implement the object aggregation method of any embodiment of the present invention.

In a fifth aspect, the embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the object aggregation method according to any embodiment of the present invention.

According to the object storage platform and the object aggregation method, device and server provided by the embodiment of the invention, the first storage pool is directly constructed on the internal storage space by adopting a solid-state disk technology, the second storage pool is constructed by adopting a conventional disk technology, the data reading performance of the first storage pool is far higher than that of the second storage pool, the first class objects which are not aggregated to the second class objects and the aggregation mapping relationship between the first class objects which are aggregated to the second class objects are stored in the first storage pool, the second class objects which are aggregated to the first class objects are stored in the second storage pool, the aggregation mapping relationship between the first class objects and the second class objects is not required to be stored by a third-party storage system, the storage structure of the first class objects is optimized, and the reading performance of the first class objects which are not aggregated to the second class objects is improved; meanwhile, the object storage gateway can regularly aggregate the first class objects which are not aggregated to the second class objects in the first storage pool into the second class objects in the second storage pool, so that dynamic aggregation between the first class objects and the second class objects is realized, excessive storage of the first class objects which are not aggregated to the second class objects in the first storage pool is prevented, and the storage performance of the first storage pool is improved.

Drawings

Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments thereof, made with reference to the following drawings:

FIG. 1 is a schematic architecture diagram of an object storage platform according to an embodiment of the present invention;

fig. 2 is a schematic structural diagram of an object storage platform according to a second embodiment of the present invention;

fig. 3 is a flowchart of an object aggregation method according to a third embodiment of the present invention;

fig. 4 is a flowchart of an object aggregation method according to a fourth embodiment of the present invention;

fig. 5 is a schematic structural diagram of an object aggregation apparatus according to a fifth embodiment of the present invention;

fig. 6 is a schematic structural diagram of a server according to a sixth embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures. In addition, the embodiments and features of the embodiments in the present invention may be combined with each other without conflict.

Example one

Fig. 1 is a schematic architecture diagram of an object storage platform according to an embodiment of the present invention. The embodiment can be applied to the condition of storing any object under the current open-source Ceph file system. Specifically, referring to fig. 1, the object storage platform 10 may include: object storage gateway 110, first storage pool 120, and second storage pool 130.

The first storage pool 120 is a storage space constructed by adopting a solid-state disk technology on an internal storage space, and stores a first class object which is not aggregated to a second class object, and an aggregation mapping relationship between the first class object which is aggregated to the second class object and the aggregated second class object, and the second storage pool 130 is a storage space constructed by adopting a conventional disk technology on the internal storage space and stores the aggregated second class object of the first class object.

Specifically, object storage gateway 110 periodically aggregates the first type objects in first storage pool 120 that are not aggregated into the second type objects in second storage pool 130, and replaces the currently aggregated first type objects in first storage pool 120 with the aggregated mapping relationship between the first type objects and the aggregated second type objects.

Optionally, because the minimum operation granularity supported by the Ceph file system is an object with a specific size, for an object with a data size higher than the minimum operation granularity, the Ceph file system can accurately execute corresponding operations such as migration or conversion, and for an object with a data size lower than the minimum operation granularity, the Ceph file system may have a risk of data loss when executing the operations such as migration or conversion, and cannot support a complete operation of the object, at this time, to solve such a problem, in this embodiment, it is proposed to aggregate a plurality of objects with a data size lower than the minimum operation granularity into an object with a data size higher than the minimum operation granularity for storage, and then execute the operations such as migration or conversion as a whole, so that the first type of object in this embodiment is an object whose operation granularity cannot support a complete operation of the object, that is, an object with a data size lower than the supported minimum operation granularity in the Ceph file system, and the second type of object is an object whose operation granularity supports a complete operation, and is a container for aggregating the first type of objects, that is a container for aggregating a plurality of objects with a data size higher than the supported by the minimum operation granularity set in the Ceph file system, and can aggregate the container; taking a media resource as an example, the first type object in this embodiment may be various types of pictures stored under the internet system, and the second type object may be a video obtained by aggregating a large number of pictures under the internet system.

At this time, in the process of aggregating the first class objects to the second class objects, the present embodiment needs to set a corresponding storage structure and an operation flow for the aggregation operation between the first class objects and the second class objects, and accurately aggregate a plurality of first class objects to a certain second class object together, so that only the storage work of the second class object needs to be followed, and the aggregated first class objects are uniformly executed through the second class objects, thereby avoiding the risk of data loss in the operation process of the first class objects, and improving the efficiency of the operations such as migration or conversion of the first class objects in the Ceph file system.

In the present embodiment, for each first-class object that has just started to be written in the Ceph file system, the first-class object may be first used as a first-class object that is not aggregated to a second-class object, and is directly stored in the first-class object 120, and then the first-class object that is not aggregated to the second-class object is periodically aggregated to the second-class object 130 through the object storage gateway 110, and the first-class object that is not aggregated to the second-class object is not aggregated to the first-class object that is not aggregated to the second-class object, so that the first-class object is not aggregated to the second-class object, and thus the first-class object that is not aggregated to the second-class object needs to be stored in the first-class object 120.

It should be noted that, in this embodiment, the object storage Gateway is mainly oriented to object storage under the Ceph file system, so the object storage Gateway 110 may be a Rados storage tier service Gateway (RGW) configured under the Ceph file system.

Meanwhile, in this embodiment, the second type objects in the second storage pool 130 may be composed of the aggregated mapping tags, attribute tags, content data, and check tags of the respective first type objects. The mapping tag is used for indicating an aggregation mapping relationship between the first class object and the second class object, where the second class object is aggregated, for example, an aggregation offset position of the aggregated first class object in the second class object, a data length of the first class object, and the like; the attribute tags are used to indicate metadata information, such as object identifiers and index classifications of each first-class object aggregated by the second-class object, at this time, the object storage platform 10 of this embodiment may further include an index record pool 140, which is used to record the index classifications of the first-class objects, for example, when each first-class object is written in the object storage platform 10, the content classification of the first-class object is firstly analyzed, and the object identifier of the first-class object is filled in under the corresponding index classification tag of the index record pool 140 to indicate that the first-class object written this time belongs to the content under the index classification tag; the content data is the actual content of each first class object aggregated by the second class object; the check label is used for judging whether each first-class object aggregated by the second-class object is aggregated to be wrong or not. When the first-class objects are aggregated into the second-class objects, the aggregated first-class objects are directly and sequentially added behind the first-class objects aggregated at the last in the second-class objects according to the data formats of the mapping tags, the attribute tags, the content data and the check tags, so that the sequence of the first-class objects aggregated by the second-class objects is ensured, and aggregation errors caused by repeated writing of the first-class objects at the same position of the second-class objects are avoided.

In addition, during the process of aggregating the first class objects into the second class objects, the second storage pool 130 is mainly responsible for storing the specific content data of the second class objects, and the second class objects are aggregated by a plurality of first class objects, at this time, the specific content data of the second class objects is the content data of each aggregated first class object, so in order to ensure the successful aggregation of the first class objects into the second class objects and prevent the excessive aggregation of the first class objects under the second class objects, this embodiment may pre-store the aggregation upper limit of the second class objects on the second storage pool 130 by analyzing the minimum operation granularity supported by the Ceph file system, so that during the process of aggregating the first class objects into the second class objects, if the aggregation occupation space of the aggregated first class objects in the second class objects reaches the aggregation upper limit, the first class objects are no longer aggregated into the second class objects, and the aggregation is switched to the next new second class objects.

For example, in the aggregation process of aggregating a first class object that is not aggregated to a second class object in the first storage pool 120 into a second class object in the second storage pool 130, before aggregating one first class object, the object storage gateway 110 first needs to detect whether an aggregation capacity of each first class object that is already aggregated under a current second class object for the current aggregation in the second storage pool 130 is greater than or equal to an aggregation upper limit of the second class object; if not, continuing to add the first class object aggregated at this time in the current second class object according to the data formats of the mapping label, the attribute label, the content data and the check label of the first class object; otherwise, the next second-class object in the second storage pool 130 is used as the new current second-class object, and the first-class objects left in the aggregation are continuously aggregated into the new current second-class object.

Further, the object storage gateway 110 may receive an operation request of the user for the first type of object stored thereon according to the read-write requirement of the user, and at this time, the object storage gateway 110 may perform corresponding processing such as reading, writing, modifying, or deleting on object information pointed by the operation request in the first storage pool 120 and the second storage pool 130 according to the operation request; therefore, if the object storage gateway 110 deletes a certain first class object according to the user requirement, and the first class object is already aggregated into a certain second class object in the second storage pool 130, the content data of the first class object needs to be correspondingly deleted in the aggregated second class object, at this time, a corresponding storage vacancy occurs in the second class object, and because the content data of different first class objects are also different, a new first class object cannot be added again at the storage vacancy position in the second class object, so that the storage space under the second class object cannot be completely used; in order to solve the above problem, in this embodiment, a reclamation process 111 is configured on the object storage gateway 110, and the recoverable capacity of each second class object is stored on the second storage pool 130, at this time, when deleting a first class object aggregated by a second class object in the second storage pool 130, only the aggregation mapping relationship between the first class object stored in the first storage pool 120 and the aggregated second class object is currently needed to be deleted, and the recoverable capacity of the second class object aggregated by the first class object deleted this time on the second storage pool 130 is modified according to the content data amount of the first class object deleted this time, and then the recoverable capacity of each second class object in the second storage pool 130 is detected in real time by the reclamation process 111, so as to timely reclaim the first class object deleted in the first class object aggregated by the second class object, thereby ensuring that the storage space of the deleted first class object can be timely reclaimed.

Specifically, the reclamation process 111 detects the recoverable capacity of each second class object in the second storage pool 130 in real time, and further finds out a target second class object whose recoverable capacity exceeds the preset reclamation upper limit, at this time, most of the first class objects aggregated by the target second class object have been deleted, at this time, because the aggregation mapping relationship between the deleted first class object and the aggregated second class object has been deleted in the first storage pool 120, each valid first class object that is not deleted is found out from each first class object aggregated by the target second class object according to the aggregation mapping relationship that is recorded in the first storage pool 120 and is related to the target second class object, and further, each valid first class object aggregated by the target second class object is rewritten into the first storage pool 120, and the aggregation mapping relationship between each valid first class object in the first storage pool 120 and the target second class object is deleted, and the target second class object in the second storage pool 130, that is migrated to the first class object aggregated by the target second class object in the first storage pool 110, and that all the storage space of the deleted first class objects aggregated first class object is deleted is wasted, and the target second class object is deleted is prevented from being deleted.

According to the technical scheme provided by the embodiment, a first storage pool is directly constructed on an internal storage space by adopting a solid-state disk technology, a second storage pool is constructed by adopting a conventional disk technology, the data reading performance of the first storage pool is far higher than that of the second storage pool, a first class object which is not aggregated to a second class object and an aggregation mapping relation between the first class object which is aggregated to the second class object and the aggregated second class object are stored in the first storage pool, the aggregated second class object of the first class object is stored in the second storage pool, and a third-party storage system is not needed to store the aggregation mapping relation between the first class object and the second class object, so that the storage structure of the first class object is optimized, and the reading performance of the first class object which is not aggregated to the second class object is improved; meanwhile, the object storage gateway can regularly aggregate the first class objects which are not aggregated to the second class objects in the first storage pool into the second class objects in the second storage pool, so that dynamic aggregation between the first class objects and the second class objects is realized, excessive storage of the first class objects which are not aggregated to the second class objects in the first storage pool is prevented, and the storage performance of the first storage pool is improved.

Example two

Fig. 2 is a schematic structural diagram of an object storage platform according to a second embodiment of the present invention. The embodiment is optimized on the basis of the technical scheme provided by the embodiment. Referring to fig. 2, the object storage platform 20 may include an object storage gateway 210, a first storage pool 220, a second storage pool 230, an index record pool 240, and a log record pool 250.

The first storage pool 220 is a storage space constructed by adopting a solid-state disk technology on an internal storage space, and stores a first class object which is not aggregated to a second class object, and an aggregation mapping relationship between the first class object which is aggregated to the second class object and the aggregated second class object, the second storage pool 230 is a storage space constructed by adopting a conventional disk technology on the internal storage space, and stores a second class object after the aggregation of the first class object, and the second storage pool 230 also stores an aggregation upper limit of the second class object; the object storage gateway 210 is configured with a recovery process 211, and the second storage pool 230 further stores the recoverable capacity of each second class object; the index record pool 240 is used for recording index classifications of the first class of objects; the log record pool 250 is used to record write logs and modification logs of the first class objects and mark corresponding aggregate checkpoint logs.

Specifically, the object storage gateway 210 periodically aggregates the first type objects in the first storage pool 220 that are not aggregated into the second type objects in the second storage pool 230, and replaces the currently aggregated first type objects in the first storage pool 220 with the aggregated mapping relationship between the first type objects and the aggregated second type objects. Meanwhile, when aggregating the first class objects in the first storage pool 220 that are not aggregated to the second class objects into the second class objects in the second storage pool 230, if the aggregation capacity of the current second class objects in the second storage pool 230 for the first class objects is greater than or equal to the aggregation upper limit, the object storage gateway 210 takes the next second class object in the second storage pool as the new current second class object, and continues to aggregate the remaining first class objects of the current aggregation into the new current second class object. In addition, the reclamation process 211 configured on the object storage gateway 210 finds, in real time, a target second class object whose recoverable capacity exceeds a preset reclamation upper limit in the second storage pool 230, rewrites, into the first storage pool 220, each valid first class object that is not deleted, from each first class object aggregated by the target second class object, and deletes an aggregation mapping relationship between each valid first class object in the first storage pool 220 and the target second class object, and the target second class object in the second storage pool 230.

Optionally, in order to explicitly indicate an aggregation process for aggregating a first class object that is not aggregated to a second class object in the first storage pool 220 into a second class object, and avoid an aggregation omission of the first class object that is not aggregated to the second class object in the first storage pool 220, in this embodiment, a corresponding search process 212 and at least one aggregation process 213 are additionally configured on the object storage gateway 210, and then the search process 212 and each aggregation process 213 cooperate together to search for the first class object that is not aggregated to the second class object in the first storage pool 220, and aggregate the searched first class object into the second class object in the second storage pool 230.

Specifically, the search process 212 configured on the object storage gateway 210 may periodically search the first class objects that are not aggregated to the second class objects in the first storage pool 220, and record the searched first class objects that are not aggregated to the second class objects one by one into the corresponding aggregation slice; a plurality of aggregation fragments are preset on the object storage gateway 210, and the aggregation fragments correspond to the aggregation processes 213 configured thereon one to one; at this time, after the search process 212 records each first-class object found from the first storage pool 220 that is not aggregated to the second-class object into each aggregation fragment, the aggregation process 213 corresponding to each aggregation fragment concurrently reads each first-class object recorded in the aggregation fragment that is not aggregated to the second-class object, and continuously aggregates the read first-class object into the second-class object in the second storage pool 230, and at the same time, replaces the first-class object aggregated this time in the first storage pool 220 with the aggregation mapping relationship between the first-class object and the aggregated second-class object, so that the content data of the first-class object aggregated to the second-class object is no longer stored in the first storage pool 220, and only the aggregation mapping relationship between the first-class object aggregated to the second-class object is stored in the first storage pool 220, thereby further ensuring the accuracy of the aggregation of the first-class object not aggregated to the second-class object in the first storage pool 220 to the second-class object 230 on the basis of avoiding excessive storage of the first storage pool 220.

Meanwhile, in order to ensure the comprehensiveness of the first type objects which are searched for in the first storage pool 220 and are not aggregated into the second type objects, the present embodiment may preset a log recording pool 250, at this time, for the newly written or modified first type objects, the present embodiment may use the newly written or modified first type objects as the first type objects which are not aggregated into the second type objects, and store the first type objects into the first storage pool 220 first, and then periodically aggregate the first type objects into the second type objects in the second storage pool 230 to ensure the read-write performance of part of the first type objects, so that when the object storage gateway 210 writes new first type objects into the first storage pool 220 or modifies the original first type objects according to the user requirements, the object storage gateway 210 may record the written log or modified log of the first type objects in the recording pool 250, so that the object storage gateway 210 periodically searches for playback logs of each first type object which are not aggregated into the second type objects by playing back the logs recorded in the log recording pool 250, and further uses the searched first type objects as the second type objects which are not aggregated into the second type objects, and sequentially plays back the second type objects into the second type objects.

Further, in order to ensure that the played logs can accurately determine whether the corresponding first-class object has been aggregated into the second-class object in the second storage pool 230, in this embodiment, when the logs recorded in the log recording pool 250 are played back periodically, the last-played log is also marked correspondingly to serve as the aggregation checkpoint log in this embodiment, at this time, because the object storage gateway 210 plays back the logs in the log recording pool 250 according to the log recording order, it can be determined that all logs in the log recording pool 250 before the marked aggregation checkpoint log have been played back, and the first-class object oriented to the played logs has been aggregated into the second-class object in the second storage pool 230 during the playback process, that is, the first type objects facing the respective write logs and modification logs in the log recording pool 250 before the marked aggregation checkpoint log are already aggregated into the second type objects in the second storage pool 230, and the first type objects facing the respective write logs and modification logs after the marked aggregation checkpoint log are not yet aggregated into the second type objects in the second storage pool 230, so that when the object storage gateway 210 searches the first type objects unaggregated into the second type objects by periodically replaying the logs in the log recording pool 250, it is only necessary to periodically replay the respective write logs and modification logs in the log recording pool 250 after the aggregation checkpoint log, at this time, each first type object of the replay facing logs is the first type object unaggregated into the second type objects in the second storage pool 230, and then sequentially aggregate each first type object facing the replay log into the second type objects in the second storage pool 230, and simultaneously according to the aggregation state, relabeling the corresponding aggregation checkpoint log in the log recording pool 250, and continuing to play back from the aggregation checkpoint log when the log is played back next time; for example, if all the logs of the current periodic playback are completely played back, the last log of the current playback is used as a new aggregation checkpoint log and is marked in the log recording pool 250, and the playback is continued from the new aggregation checkpoint log next time, so that the efficiency and the accuracy of searching for the first-class objects which are not aggregated into the second-class objects are improved.

For example, the present embodiment takes the example that the search process 212 and the at least one aggregation process 213 configured on the object storage gateway 210 cooperate to determine each first type object that is not aggregated to the second type object by periodically playing back each write log and modification log located after the aggregation checkpoint log in the log recording pool 250, and a specific search process of each first type object that is not aggregated to the second type object in the first storage pool 220 is described in detail:

a search process 212 configured on the object storage gateway 210 may periodically play back each write log and modification log located after the aggregation checkpoint log in the log recording pool 250, and correspondingly write each playback log into each aggregation slice, at this time, each aggregation process 213 concurrently reads each playback log from each corresponding aggregation slice, and aggregates the first-class object facing each playback log as the first-class object that is not aggregated to the second-class object into the second-class object in the second storage pool 230, at this time, in the aggregation process of each first-class object, the aggregation process 213 first detects whether the aggregation capacity of each first-class object that has been aggregated under the current second-class object for the current aggregation in the second storage pool 230 is greater than or equal to the aggregation upper limit of the second-class object; if not, continuing to add the first class object aggregated at this time in the current second class object according to the data formats of the mapping label, the attribute label, the content data and the check label of the first class object; otherwise, the next second-class object in the second storage pool 230 is used as the new current second-class object, and the first-class objects left in the aggregation are continuously aggregated into the new current second-class object.

In addition, in this embodiment, there may be various operations such as writing, reading, deleting, and modifying for the first class object in the object storage platform 20 according to the user requirement, and the following explains each operation process of the first class object in this embodiment in detail:

1) For the write operation of the first class object, in order to ensure the read-write performance of part of the first class object and avoid excessive storage of the first storage pool 220, in this embodiment, for the preliminary write of each first class object, the newly written first class object is directly used as the first class object that is not aggregated into the second class object, and the content data of the first class object is directly stored into the first storage pool 220, and then the object storage gateway 210 periodically aggregates each first class object that is not aggregated into the second class object in the second storage pool 230, so as to further improve the read-write performance of the first class object in the preliminary write stage on the basis of avoiding excessive storage of the first storage pool 220.

Specifically, if the object storage gateway 210 receives a write request for a first class object, it first takes the first class object written this time as a first class object that is not aggregated to a second class object, directly stores the first class object in the first storage pool 220, and correspondingly records a write log of the first class object written this time in the log recording pool 250, so as to sequentially play back each write log and modification log located after the aggregation checkpoint log in the log recording pool 250 at regular intervals, at this time, the write log of the first class object written this time will be played back, and then, the first class object written this time is aggregated to the second class object in the second storage pool 230.

2) For the reading operation of the first type object, since the first type object may be stored in the first storage pool 220, or may be stored in the second type object in the second storage pool 230, in order to ensure the accuracy of the read first type object, this embodiment first reads the content data of the first type object read this time in the first storage pool 220; if the content data read from the first storage pool 220 is empty, the aggregation mapping relationship between the first class objects read this time and the aggregated second class objects is continuously found from the first storage pool 220, the second class objects aggregated from the first class objects read this time are found from the second storage pool 230 according to the aggregation mapping relationship, and the content data of the first class objects read this time is continuously read from the second class objects.

Specifically, if the object storage gateway 210 receives a reading request of a first class object, because the first class object that is not aggregated to the second class object and the aggregation mapping relationship between the first class object that is aggregated to the second class object and the aggregated second class object are stored in the first storage pool 220, first, the content data of the first class object that is read this time is read in the first storage pool 220, and if the content data of the first class object can be read, it indicates that the first class object that is read this time is not aggregated to the second class object in the second storage pool 230, and then the read content data is directly fed back to the user; on the other hand, if the content data read from the first storage pool 220 is empty, it indicates that the first class object read this time is already aggregated into the second class object in the second storage pool 230, so it is necessary to find out the aggregation mapping relationship between the first class object read this time and the aggregated second class object in the first storage pool 220, such as the offset position of the first class object in the aggregated second class object and the data length of the first class object, and further find out the second class object aggregated by the first class object read this time in the second storage pool 230 according to the aggregation mapping relationship, and continue to read the content data with the corresponding data length at the corresponding offset position of the second class object, as the content data of the first class object read this time. In addition, in order to ensure the accuracy of the read content data, the embodiment may also read the check tag of the first-class object, and determine whether the content data read this time is erroneous by using the check tag, so as to improve the reading accuracy of the first-class object.

3) For the deletion operation of the first class object, since the first class object may be stored in the first storage pool 220 or the second class object in the second storage pool 230, it needs to first determine whether the first class object deleted this time is the first class object that is not aggregated to the second class object or the first class object that is aggregated to the second class object, so as to perform different deletion operations in the following, thereby ensuring the accuracy of the deletion this time.

Specifically, if the object storage gateway 210 receives a deletion request of a first class object and the first class object deleted this time is a first class object that is not aggregated to a second class object, it indicates that the first class object read this time is stored in the first storage pool 220, and at this time, a write log or a modification log of the first class object deleted this time is recorded in the log recording pool 250, so that the first class object can be aggregated into the second class object when the log is played back later on regularly, and therefore, in this embodiment, the content data of the first class object deleted this time is directly deleted in the first storage pool 220, and meanwhile, in order to avoid aggregation error caused by aggregation of the first class object deleted this time when the log is played back later on a regular basis, the write log or the modification log of the first class object deleted this time is also deleted in the log recording pool 250, so that the write log or the modification log of the first class object deleted this time is not played back later, thereby ensuring the aggregation accuracy of the first class object. However, if the object storage gateway 210 receives a deletion request for a first class object, and the deleted first class object is a first class object aggregated to a second class object, it indicates that the deleted first class object is stored in the second class object in the second storage pool 230, at this time, a write log or a modification log of the deleted first class object is played back in the log recording pool 250, and the playback is not performed again, so only the aggregated mapping relationship between the deleted first class object and the aggregated second class object in the first storage pool 220 needs to be deleted, and the content data of the first class object does not need to be deleted in the second class object aggregated to the deleted first class object in the second storage pool 230, a recovery process 211 configured on the object storage gateway 210 detects the recoverable capacity of each second class object in the second storage pool 230 to perform a corresponding recovery operation, at this time, in order to ensure the recovery accuracy, only the recoverable capacity of the first class object newly deleted in the second storage pool 230 is added to the recoverable capacity of each second class object, and the recoverable object exceeds the predetermined recoverable target capacity of the first class object in the second storage pool 230, and the recoverable object exceeds the predetermined target recovery limit, and the recoverable first class object in the second storage pool 230.

4) For the modification operation of the first class object, the modification operation in this embodiment is mainly to modify the content data of the first class object that has been written currently, and since the first class object may be stored in the first storage pool 220, and may also be stored in the second class object in the second storage pool 230, it needs to be determined whether the first class object modified this time is the first class object that is not aggregated to the second class object, or the first class object that is aggregated to the second class object, so as to perform different modification operations subsequently, and ensure the accuracy of the modification this time; if the first class object modified this time is already aggregated in the second class object in the second storage pool 230, and the content length before and after modification is different, the content data of the first class object modified this time cannot be modified in the second class object, so the modification operation can be used as a combination of a deletion operation and a write operation, that is, the object storage gateway 210 executes a corresponding deletion operation on the written original first class object, and after the deletion is successful, executes a corresponding write operation on the new first class object which is currently required to be written again, thereby ensuring the accurate execution of the modification operation.

Specifically, if the object storage gateway 210 receives a modification request for a first class object, and the first class object modified this time is a first class object that is not aggregated to a second class object, it indicates that the first class object read this time is stored in the first storage pool 220, and at this time, a write log of the first class object deleted this time is recorded in the log recording pool 250, so that the content data of the first class object modified this time in the first storage pool 220 may be directly updated, and meanwhile, to avoid aggregation error caused by aggregation of the content data initially written in the first class object modified this time during subsequent regular log playback, a modification log of the first class object modified this time may also be recorded in the log recording pool 250, and the write log of the first class object modified this time may be deleted, so that only the modification log of the first class object modified this time is played back later, and the write log before modification of the first class object modified this time is no longer played back, thereby ensuring accuracy of the aggregated content data during subsequent aggregation of the first class object modified this time. However, if the object storage gateway 210 receives a modification request for a first class object and the first class object modified this time is a first class object aggregated to a second class object, it indicates that the first class object modified this time is stored in the second class object in the second storage pool 230, and at this time, the write log of the first class object modified this time has already been played back in the log recording pool 250, and is not played back again, so only the first class object modified this time needs to be directly written in the first storage pool 220, at this time, because when reading the first class object, corresponding content data is first read in the first storage pool 220, and if the content data can be read, the aggregation mapping relationship between the first class object and the second class object will not be concerned any more, therefore, even if the first class object modified this time is written in the first storage pool 220, when the aggregation mapping relationship between the content data of the first-class object, which is stored in the first storage pool 220 before modification, and the aggregated second-class object is not deleted, the reading accuracy of the first-class object modified this time can be ensured, so when the first-class object modified this time is written in the first storage pool 220, the aggregation mapping relationship between the content data of the first-class object, which is stored in the first storage pool 220 before modification, and the aggregated second-class object is not required to be deleted, but the modification log of the first-class object modified this time needs to be recorded in the log recording pool 250, so that when the log is played back later regularly, the first-class object modified this time can be re-aggregated in the second-class object in the second storage pool 230 according to the modification log, at this time, when the first-class object modified this time is aggregated in the second-class object, the aggregation mapping relationship between the first class objects before the modification and the aggregated second class objects needs to be deleted in the first storage pool 220, and the aggregation mapping relationship between the first class objects after the modification and the aggregated second class objects is further stored in the first storage pool 220, so that the reading accuracy of the first class objects after the modification is ensured.

According to the technical scheme provided by the embodiment, the corresponding aggregation checkpoint logs are marked in the log recording pool, the first class objects which are not aggregated to the second class objects are accurately distinguished from the first class objects which are aggregated to the second class objects, the write logs and the modification logs which are positioned behind the aggregation checkpoint logs in the log recording pool are played back at regular intervals, and the first class objects facing each playback log are directly used as the first class objects which are not aggregated to the second class objects to be aggregated to the second class objects in the second storage pool, so that the accuracy of aggregation of the first class objects to the second class objects is ensured, dynamic aggregation between the first class objects and the second class objects is realized, excessive storage of the first class objects which are not aggregated to the second class objects in the first storage pool is prevented, and the storage performance of the first class objects is improved.

EXAMPLE III

Fig. 3 is a flowchart of an object aggregation method provided in the third embodiment of the present invention, and this embodiment is applicable to the object storage platform provided in the foregoing embodiment in a case where any object is stored in the current open-source Ceph file system. The object aggregation method provided by this embodiment may be executed by an object aggregation apparatus provided by the embodiment of the present invention, and the apparatus may be implemented by software and/or hardware and integrated in a server executing the method.

Specifically, referring to fig. 3, the method may include the steps of:

s310, the first class objects which are not aggregated to the second class objects are found out in the first storage pool regularly.

The first storage pool is a storage space constructed by adopting a solid-state disk technology on an internal storage space, and stores a first class object which is not aggregated to a second class object, and an aggregation mapping relation between the first class object which is aggregated to the second class object and the aggregated second class object, and the second storage pool is a storage space constructed by adopting a conventional disk technology on the internal storage space and stores a second class object which is aggregated by the first class object.

Specifically, in this embodiment, for each first class object that is just written into the Ceph file system, the first class object is first regarded as a first class object that is not aggregated to the second class object, and is directly stored in the first storage pool, so that in this embodiment, by analyzing the storage condition of each first class object in the first storage pool through the object storage gateway, the first class object that is newly written into the object storage platform in the first storage pool and is not aggregated to the second class object in the second storage pool is periodically found out, so as to be accurately aggregated to the second class object in the second storage pool in the subsequent time.

For example, in order to ensure the comprehensiveness of the first class object which is found in the first storage pool and is not aggregated into the second class object, as for the newly written or modified first class object, the first class object is taken as the first class object which is not aggregated into the second class object, and is stored in the first storage pool first, and then is periodically aggregated into the second class object in the second storage pool to ensure the read-write performance of part of the first class object, a log recording pool is preset in the embodiment, when a new first class object is written into the first storage pool or an original first class object is modified according to a user requirement, a write log or a modified log of the first class object is recorded in the log recording pool, so that the object storage gateway plays back logs recorded in the log recording pool periodically, finds a playback log of each first class object which is not aggregated into the second class object, and then sequentially aggregates the found playback log into the second class object in the second storage pool as the first class object which is not aggregated into the second class object.

At this time, in this embodiment, periodically finding out the first class object that is not aggregated to the second class object in the first storage pool may specifically include: periodically playing back each write log and modification log which are positioned behind the aggregation checkpoint log in the log recording pool; and taking the first-class objects oriented by the playback logs as the first-class objects which are not aggregated to the second-class objects.

Specifically, in order to ensure that the played back log can accurately determine whether the corresponding first class object is aggregated into the second class object in the second storage pool, in this embodiment, when playing back each log recorded in the log recording pool regularly, the last played back log is also marked correspondingly, as the aggregated checkpoint log in this embodiment, at this time, because the object storage gateway plays back the logs in the log recording pool according to the log recording sequence, it can be determined that the logs located before the marked aggregated checkpoint log in the log recording pool have all been played back, the first class object oriented to the played back log is aggregated into the second class object in the second storage pool during the playback process, that is, each written log located before the marked aggregated checkpoint log in the log recording pool and the first class object oriented to the modified log are aggregated into the second class object in the second storage pool, and each written log located after the marked aggregated checkpoint log and the first class object oriented to the first class object not aggregated into the second storage pool only need to be played back by the aggregated checkpoint object, and when each written log and each modified log located after the marked aggregated checkpoint log are sequentially stored into the second class object in the second storage pool, the aggregated checkpoint object is only needed to be played back by the aggregated object oriented to the aggregated first class object, and each log storage pool, and thus each log is recorded into the playback object in the second class object oriented to which is not aggregated checkpoint object in the aggregated into the second storage pool; meanwhile, after aggregating the first class objects which are not aggregated to the second class objects into the second class objects in the second storage pool, the method may further include: according to the aggregation state, the aggregation check point logs are marked again in each written log and each modified log in the log record pool, so that the logs are played back from the aggregation check point logs continuously when the logs are played back next time; for example, if all the logs of the current periodic playback are completely played back, the last log of the current playback is used as a new aggregation check point log, the log is marked in the log recording pool, and the playback is continued from the new aggregation check point log next time, so that the efficiency and the accuracy of searching the first-class objects which are not aggregated to the second-class objects are improved.

And S320, aggregating the first class objects which are not aggregated to the second class objects into the second class objects in the second storage pool, and replacing the aggregated first class objects in the first storage pool with the aggregated mapping relation between the first class objects and the aggregated second class objects.

Specifically, after the first class objects which are not aggregated to the second class objects are regularly found in the first storage pool, each found first class object is continuously aggregated to the second class objects in the first storage pool, and at this time, the first class objects aggregated this time may be directly added to the first class objects aggregated last in the second class objects according to the data formats of the mapping tag, the attribute tag, the content data, and the check tag, so as to ensure the order of the first class objects aggregated by the second class objects, and avoid aggregation errors caused by repeatedly writing the first class objects at the same position of the second class objects.

In addition, in the process of aggregating the first type objects to the second type objects, the second storage pool is mainly responsible for storing specific content data of the second type objects, and the second type objects are aggregated by a plurality of first type objects, and at this time, the specific content data of the second type objects is content data of each aggregated first type object, so in order to ensure successful aggregation of the first type objects to the second type objects and prevent excessive aggregation of the first type objects under the second type objects, this embodiment may store an aggregation upper limit of the second type objects in advance on the second storage pool by analyzing a minimum operation granularity supported by the Ceph file system, and at this time, aggregate the first type objects that are not aggregated to the second type objects into the second type objects in the second storage pool, which may specifically include: for each first class object which is not aggregated to the second class object in the first storage pool, if the aggregation capacity of the current second class object to the first class object in the second storage pool is smaller than the aggregation upper limit, directly adding the first class object in the current second class object; otherwise, taking the next second class object in the second storage pool as a new current second class object, and adding the first class object in the new current second class object; in the process of aggregating the first class objects to the second class objects, if the aggregated occupied space of the aggregated first class objects in the second class objects reaches the aggregation upper limit, the first class objects are not aggregated into the second class objects any more, and the aggregation is switched to the next new second class objects.

For example, in an aggregation process of aggregating a first class of objects, which are not aggregated to a second class of objects in a first storage pool, to a second class of objects in a second storage pool, an object storage gateway first needs to detect whether an aggregation capacity of each aggregated first class of objects in the second storage pool for the current aggregation of the second class of objects is greater than or equal to an aggregation upper limit of the second class of objects; if the current second class object is smaller than the aggregation upper limit, continuing to add the first class object aggregated at this time in the current second class object according to the data formats of the mapping label, the attribute label, the content data and the check label of the first class object; otherwise, taking the next second class object in the second storage pool as a new current second class object, and continuing to aggregate the first class objects left in the current aggregation into the new current second class object.

Meanwhile, in order to explicitly indicate an aggregation process of aggregating a first class object, which is not aggregated to a second class object, in a first storage pool into the second class object, and avoid aggregation omission of the first class object, which is not aggregated to the second class object, in the first storage pool, the first class object, which is not aggregated to the second class object, is periodically searched in the first storage pool through a search process configured on the object storage gateway, and the searched first class objects, which are not aggregated to the second class object, are recorded into corresponding aggregation fragments one by one; the object storage gateway is preset with a plurality of aggregation fragments, and the aggregation fragments correspond to aggregation processes configured on the object storage gateway one by one; at this time, after the search process records each first-class object which is found from the first storage pool and is not aggregated to the second-class object to each aggregation fragment, the aggregation process corresponding to each aggregation fragment concurrently reads each first-class object which is recorded in the aggregation fragment and is not aggregated to the second-class object, and continuously aggregates the read first-class object to the second-class object in the second storage pool, and at the same time, the aggregated first-class object in the first storage pool is replaced with the aggregation mapping relationship between the first-class object and the aggregated second-class object, so that the content data of the first-class object aggregated to the second-class object is no longer stored in the first storage pool, and only the aggregation mapping relationship between the first-class object aggregated to the second-class object and the aggregated second-class object is stored, thereby, on the basis of avoiding excessive storage of the first storage pool, further ensuring the accuracy of aggregating the first-class object which is not aggregated to the second-class object in the first storage pool to the second-class object in the second storage pool.

For example, in this embodiment, by taking as an example that each write log and modification log located after the aggregation checkpoint log in the log recording pool are periodically played back, and a search process configured on the object storage gateway and at least one aggregation process cooperate together to determine each first class object that is not aggregated to the second class object, a specific search process of each first class object that is not aggregated to the second class object in the first storage pool is described in detail:

optionally, a search process configured on the object storage gateway may periodically play back each written log and modified log located after the aggregation checkpoint log in the log recording pool, and correspondingly write each playback log into each aggregation fragment, at this time, each aggregation process concurrently reads each playback log from each corresponding aggregation fragment, and aggregates a first class object facing each playback log as a first class object that is not aggregated to a second class object into a second class object in the second storage pool, and during an aggregation process of each first class object, the aggregation process first detects whether an aggregation capacity of each first class object aggregated under a current second class object for the current aggregation in the second storage pool is greater than or equal to an aggregation upper limit of the second class object; if not, continuing to add the aggregated first class object in the current second class object according to the data formats of the mapping label, the attribute label, the content data and the check label of the first class object; otherwise, taking the next second class object in the second storage pool as a new current second class object, and continuing to aggregate the first class objects left in the current aggregation into the new current second class object.

Example four

Fig. 4 is a flowchart of an object aggregation method according to a fourth embodiment of the present invention, and the present embodiment is optimized based on the foregoing embodiments. Specifically, as shown in fig. 4, this embodiment mainly explains in detail various operation processes of the first class object and a recycling process of the second class object in the process of aggregating the first class object to the second class object.

Specifically, referring to fig. 4, the method may include the steps of:

s410, the first class objects which are not aggregated to the second class objects are found out in the first storage pool regularly.

And S420, aggregating the first class objects which are not aggregated to the second class objects into the second class objects in the second storage pool, and replacing the first class objects aggregated this time into the aggregation mapping relationship between the first class objects and the aggregated second class objects in the first storage pool.

S440, detecting the recoverable capacity of each second class object in the second storage pool in real time, rewriting each effective first class object aggregated by the target second class object with the recoverable capacity exceeding the preset recovery upper limit into the first storage pool, and deleting the aggregation mapping relation between each effective first class object and the target second class object in the first storage pool and the target second class object in the second storage pool.

Optionally, according to the read-write requirement of the user, an operation request of the user for the first type of object stored thereon may be received, and at this time, the object storage gateway may perform, according to the operation request, corresponding processing of reading, writing, modifying, or deleting, etc., on object information pointed by the operation request in the first storage pool and the second storage pool; therefore, if the object storage gateway deletes a certain first class object according to the user requirement, and the first class object is already aggregated into a certain second class object in the second storage pool, the content data of the first class object needs to be correspondingly deleted in the aggregated second class object, at this time, a corresponding storage vacancy occurs in the second class object, and because the content data of different first class objects are also different, a new first class object cannot be added again at the storage vacancy position in the second class object, so that the storage space under the second class object cannot be completely used; in order to solve the above problem, in this embodiment, a recovery process is configured on the object storage gateway, and the recoverable capacity of each second class object is stored in the second storage pool, at this time, when deleting the first class object aggregated by the second class objects in the second storage pool, only the aggregation mapping relationship between the first class object stored in the first storage pool and the aggregated second class object is currently needed to be deleted, and the recoverable capacity of the second class object aggregated by the first class object deleted this time on the second storage pool is modified according to the content data amount of the first class object deleted this time, and then the recoverable capacity of each second class object in the second storage pool is detected in real time by the recovery process, so as to timely recover the first class object deleted in the first class object aggregated by the second class object, thereby ensuring that the storage space of the deleted first class object can be timely recovered.

Specifically, the recoverable capacity of each second class object in the second storage pool is detected in real time through a recovery process, and then a target second class object of which the recoverable capacity exceeds a preset recovery upper limit is found out, at this time, most of the first class objects aggregated by the target second class object are deleted, at this time, because the aggregation mapping relationship between the deleted first class object and the aggregated second class object is deleted in the first storage pool, each valid first class object which is not deleted is found out from each first class object aggregated by the target second class object according to the aggregation mapping relationship which is recorded in the first storage pool and is related to the target second class object, and then each valid first class object aggregated by the target second class object is rewritten into the first storage pool, the aggregation mapping relationship between each valid first class object in the first storage pool and the target second class object is deleted, and the target second class object in the second storage pool is deleted, that all valid first class objects aggregated by the target second class object are migrated into the first storage pool again, and all the deleted first class objects in the second storage pool are deleted, so that the aggregate first class objects aggregated by the target second class object are deleted can be recovered, and thus it is avoided that all the storage objects aggregated by the target second class objects are deleted.

For example, in this embodiment, rewriting each valid first class object aggregated by the target second class object whose recoverable capacity exceeds the preset recovery upper limit into the first storage pool may specifically include: searching an aggregation mapping relation between each first class object aggregated by the target second class object and the target second class object in the first storage pool respectively; and rewriting the first class objects with the non-empty aggregation mapping relationship into the first storage pool as the valid first class objects aggregated by the target second class objects. Determining each first class object aggregated by the target second class objects in sequence, and judging whether an aggregation mapping relationship between the first class object and the target second class object exists in the first storage pool, if not, indicating that the first class object is deleted without any treatment, otherwise, determining that the first class object with the aggregation mapping relationship being non-empty is the first class object which is not deleted in the target second class object, rewriting the first class object which is taken as an effective first class object aggregated by the target second class object into the first storage pool, deleting the aggregation mapping relationship between each effective first class object in the first storage pool and the target second class object, and deleting the target second class object in the second storage pool.

S440, if the operation request of the first class object is received, the object information pointed under the first storage pool and the second storage pool is updated correspondingly.

Optionally, there may be operations such as writing, reading, deleting, and changing corresponding to the stored first class object according to a user requirement, in order to ensure that the information stored in the first storage pool and the second storage pool is dynamically updated under the operations, in this embodiment, it may be detected in real time by the object storage gateway whether a user has a certain operation requirement on the first class object, if an operation request for the first class object is received, the object information pointed by the operation request is directly searched in the first storage pool and the second storage pool, at this time, the object information may be content data of the first class object that is not aggregated to the second class object in the first storage pool or an aggregation mapping relationship between the first class object that is aggregated to the second class object and the aggregated second class object in the first storage pool, or content data of the first class object that is aggregated to the second class object in the second storage pool, and the object information pointed by the operation request that is searched in the first storage pool and the second storage pool is updated in association with the current operation, so as to ensure that the dynamic update of the object information in the first storage pool and the second storage pool is updated, thereby improving accuracy of the operation object.

For example, the present embodiment may have operations such as writing, reading, deleting, and modifying corresponding to the stored first type object according to a user requirement, and each operation process of the first type object in the present embodiment is explained in detail as follows:

1) If a writing request of the first class object is received, the first class object written at this time is directly stored in a first storage pool as the first class object which is not aggregated to the second class object, and a writing log of the first class object written at this time is recorded in a log recording pool.

Specifically, if a write request for a first class object is received through the object storage gateway, the first class object written this time is first used as the first class object which is not aggregated to a second class object, and is directly stored in the first storage pool, and a write log of the first class object written this time is correspondingly recorded in the log recording pool, so that each write log and a modification log which are located after an aggregation check point log in the log recording pool are played back at regular intervals, at this time, the write log of the first class object written this time is played back, and then the first class object written this time is aggregated to the second class object in the second storage pool.

2) If a reading request of the first class object is received, reading the content data of the first class object read at this time in a first storage pool; and if the content data read out from the first storage pool is empty, finding out an aggregation mapping relation between the first class objects read out this time and the aggregated second class objects in the first storage pool, and continuously reading the content data of the first class objects read out this time in the second class objects aggregated by the first class objects read out this time in the second storage pool according to the aggregation mapping relation.

Specifically, if a reading request of a first class object is received through an object storage gateway, because the first class object which is not aggregated to a second class object and an aggregation mapping relationship between the first class object which is aggregated to the second class object and the aggregated second class object are stored in a first storage pool, firstly reading content data of the first class object which is read at this time in the first storage pool, and if the content data of the first class object can be read, indicating that the first class object which is read at this time is not aggregated to the second class object in the second storage pool, and further directly feeding back the read content data to a user; if the content data read out from the first storage pool is empty, it indicates that the first class of objects read out this time are already aggregated into the second class of objects in the second storage pool, so that it is necessary to find out the aggregation mapping relationship between the first class of objects read out this time and the aggregated second class of objects in the first storage pool, such as the offset position of the first class of objects in the aggregated second class of objects, the data length of the first class of objects, and the like, further find out the second class of objects aggregated by the first class of objects read out this time in the second storage pool according to the aggregation mapping relationship, and continue to read out the content data with the corresponding data length at the corresponding offset position of the second class of objects as the content data of the first class of objects read out this time.

3) If a deleting request of a first class object is received, and the deleted first class object is a first class object which is not aggregated to a second class object, deleting the deleted first class object in a first storage pool, and deleting a write log of the deleted first class object in a log recording pool; if a deletion request of the first class object is received, and the deleted first class object is the first class object which is aggregated to the second class object, deleting the aggregation mapping relation between the deleted first class object and the aggregated second class object in the first storage pool, and updating the recoverable capacity of the aggregated second class object of the deleted first class object in the second storage pool.

Specifically, if a deletion request of a first class object is received through the object storage gateway, and the first class object deleted this time is a first class object that is not aggregated to a second class object, it indicates that the first class object read this time is stored in the first storage pool, and at this time, a write log or a modification log of the first class object deleted this time is recorded in the log recording pool, so that the first class object can be aggregated into the second class object when logs are played back later regularly. However, if a delete request of a first class object is received through the object storage gateway, and the deleted first class object is a first class object aggregated to a second class object, it indicates that the deleted first class object is stored in the second class object in the second storage pool, and at this time, a write log or a modification log of the deleted first class object is already played back in the log record pool, and the object is not played back again, so only the aggregation mapping relationship between the deleted first class object and the aggregated second class object needs to be deleted in the first storage pool, and it is not necessary to delete content data of the first class object in the aggregated second class object of the deleted first class object in the second storage pool, a recovery process configured on the object storage gateway detects a recoverable capacity of each second class object in the second storage pool to perform a corresponding recovery operation, and at this time, in order to ensure recovery accuracy, only the recoverable capacity of the aggregated second class object aggregated to the deleted first class object in the second storage pool needs to be updated in the second storage pool, and the recoverable capacity of each second class object exceeds a preset upper limit, and the recoverable data of each second class object exceeds a preset recoverable target storage pool.

4) If a modification request of a first class object is received and the currently modified first class object is the first class object which is not aggregated to a second class object, updating the content data of the currently modified first class object in a first storage pool, simultaneously recording a modification log of the currently modified first class object in a log recording pool, and deleting a write log of the currently modified first class object; if a modification request of a first class object is received, and the first class object modified this time is a first class object aggregated to a second class object, the modified first class object is directly written in a first storage pool, and when the modified first class object is aggregated to the second class object, an aggregation mapping relation between the first class object before modification this time and the aggregated second class object is deleted in the first storage pool.

Specifically, if a modification request of a first class object is received through the object storage gateway, and the first class object modified this time is a first class object that is not aggregated to a second class object, it indicates that the first class object read this time is stored in the first storage pool, and at this time, a write log of the first class object deleted this time is recorded in the log recording pool, so that content data of the first class object modified this time in the first storage pool can be directly updated, and meanwhile, in order to avoid aggregation error caused by aggregation of content data initially written in the first class object modified this time when logs are subsequently played back regularly, a modification log of the first class object modified this time is recorded in the log recording pool, and the write log of the first class object modified this time is deleted, so that only the modification log of the first class object modified this time is played back later, and the write log before the first class object modified this time is not played back any more, thereby ensuring accuracy of the aggregated content data of the first class object modified this time in subsequent aggregation. However, if a modification request of a first class object is received through the object storage gateway, and the first class object modified this time is a first class object aggregated to a second class object, it indicates that the first class object modified this time is stored in the second class object in the second storage pool, and at this time, a write log of the first class object modified this time has already been played back in the log recording pool, and is not played back again, so only the first class object modified this time needs to be directly written in the first storage pool, at this time, because when the first class object is read, corresponding content data will be read in the first storage pool first, if the content data can be read, the aggregation mapping relationship between the first class object and the second class object will not be concerned any more, therefore, even if the first class object modified this time is written in the first storage pool, when the aggregated mapping relation between the content data of the first-class object before modification and the aggregated second-class object stored in the first storage pool is not deleted, the reading accuracy of the modified first-class object can be ensured, so that when the modified first-class object is written into the first storage pool, the aggregated mapping relation between the content data of the first-class object before modification and the aggregated second-class object stored in the first storage pool is not required to be deleted, but the modification log of the modified first-class object is required to be recorded into the log recording pool, so that when the log is played back later, the modified first-class object can be re-aggregated into the second-class object in the second storage pool according to the modification log, and at the moment, when the modified first-class object is aggregated into the second-class object, the aggregated first-class object before modification and the aggregated second-class object aggregated into the first storage pool are required to be deleted And the mapping relation is obtained, and the aggregation mapping relation between the modified first-class object and the aggregated second-class object is stored in the first storage pool, so that the reading accuracy of the modified first-class object is ensured.

According to the technical scheme provided by the embodiment, a first storage pool is directly constructed on an internal storage space by adopting a solid-state disk technology, a second storage pool is constructed by adopting a conventional disk technology, the data reading performance of the first storage pool is far higher than that of the second storage pool, a first class object which is not aggregated to a second class object and an aggregation mapping relation between the first class object which is aggregated to the second class object and the aggregated second class object are stored in the first storage pool, the aggregated second class object of the first class object which is aggregated to the second class object is stored in the second storage pool, a third-party storage system is not required to store the aggregation mapping relation between the first class object and the second class object, the storage structure of the first class object is optimized, and the reading performance of the first class object which is not aggregated to the second class object is improved; meanwhile, the object storage gateway can regularly aggregate the first class objects which are not aggregated to the second class objects in the first storage pool into the second class objects in the second storage pool, so that dynamic aggregation between the first class objects and the second class objects is realized, excessive storage of the first class objects which are not aggregated to the second class objects in the first storage pool is prevented, and the storage performance of the first storage pool is improved; meanwhile, the recoverable capacity of each second type object in the second storage pool is detected in real time through the recovery process, the storage space of the deleted first type object can be ensured to be recovered in time, and the waste of the storage space on the second type object aggregated by the deleted first type object is avoided.

EXAMPLE five

Fig. 5 is a schematic structural diagram of an object aggregation apparatus according to a fifth embodiment of the present invention, which is disposed in an object storage platform according to the fifth embodiment of the present invention, specifically, as shown in fig. 5, the apparatus may include:

an object searching module 510, configured to periodically search the first storage pool for a first class of objects that are not aggregated to a second class of objects;

an object aggregation module 520, configured to aggregate a first class object that is not aggregated to a second class object into the second class object in the second storage pool, and replace the aggregated first class object in the first storage pool with an aggregated mapping relationship between the first class object and the aggregated second class object;

The object aggregation device provided by the embodiment can be applied to the object aggregation method provided by any of the above embodiments, and has corresponding functions and beneficial effects.

EXAMPLE six

Fig. 6 is a schematic structural diagram of a server according to a sixth embodiment of the present invention, as shown in fig. 6, the server includes a processor 60, a storage device 61, and a communication device 62; the number of the processors 60 in the server may be one or more, and one processor 60 is taken as an example in fig. 6; the processor 60, the storage device 61 and the communication device 62 in the server may be connected by a bus or other means, and the bus connection is taken as an example in fig. 6.

The server provided by this embodiment can be used to execute the object aggregation method provided by any of the above embodiments, and has corresponding functions and advantages.

EXAMPLE seven

An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, can implement the object aggregation method in any of the embodiments. The method specifically comprises the following steps:

periodically finding out the first class objects which are not aggregated to the second class objects in the first storage pool;

the first storage pool is a storage space constructed by adopting a solid-state disk technology on an internal storage space, and stores a first class object which is not aggregated to a second class object and an aggregation mapping relation between the first class object which is aggregated to the second class object and the aggregated second class object, and the second storage pool is a storage space constructed by adopting a conventional disk technology on the internal storage space and stores a second class object aggregated by the first class object.

Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the method operations described above, and may also perform related operations in the object aggregation method provided by any embodiment of the present invention.

From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly can be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention or portions thereof contributing to the prior art may be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.

It should be noted that, in the embodiment of the object aggregation apparatus, the included units and modules are merely divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made to the present invention by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. An object storage platform, comprising: the system comprises an object storage gateway, a first storage pool and a second storage pool, wherein the first storage pool is a storage space constructed by adopting a solid-state disk technology on an internal storage space, stores a first class object which is not aggregated to a second class object, and an aggregation mapping relation between the first class object which is aggregated to the second class object and the aggregated second class object, and the second storage pool is a storage space constructed by adopting a conventional disk technology on the internal storage space and stores the second class object aggregated by the first class object; wherein the content of the first and second substances,

and the object storage gateway periodically aggregates the first class objects which are not aggregated to the second class objects in the first storage pool into the second class objects in the second storage pool, and replaces the aggregated first class objects in the current time into an aggregation mapping relation between the first class objects and the aggregated second class objects in the first storage pool.

2. The object storage platform of claim 1, wherein the second class of objects in the second storage pool is composed of the aggregated mapping tags, attribute tags, content data, and check tags of the respective first class of objects.

3. The object storage platform of claim 1, wherein the object storage gateway is configured with a lookup process and at least one aggregation process; wherein the content of the first and second substances,

the searching process searches for the first class objects which are not aggregated to the second class objects in the first storage pool regularly, and records the first class objects in the corresponding aggregation fragments;

and the aggregation process reads the first class objects which are recorded in the corresponding aggregation fragments and are not aggregated to the second class objects concurrently, aggregates the read first class objects to the second class objects in the second storage pool, and simultaneously replaces the first class objects aggregated this time with the aggregation mapping relationship between the first class objects and the aggregated second class objects in the first storage pool.

4. The object storage platform of claim 1, further comprising a log record pool for recording write logs and modification logs of the first class of objects and marking corresponding aggregate checkpoint logs;

the object storage gateway periodically plays back each written log and modified log which are positioned behind the aggregation check point log in the log recording pool, takes the first class object facing each played log as the first class object which is not aggregated to the second class object, and re-marks the aggregation check point log in the log recording pool according to the aggregation state.

5. The object storage platform of claim 1, wherein the second storage pool further records an aggregate upper bound for objects of the second type;

when the object storage gateway aggregates the first class objects which are not aggregated to the second class objects in the first storage pool into the second class objects in the second storage pool, if the aggregation capacity of the current second class objects in the second storage pool to the first class objects is larger than or equal to the aggregation upper limit, the next second class object in the second storage pool is used as a new current second class object, and the rest of the first class objects aggregated at this time are continuously aggregated into the new current second class object.

6. The object storage platform of any one of claims 1-5, wherein a reclamation process is configured on the object storage gateway, and wherein the second storage pool further records a reclaimable volume for each object of the second type; wherein the content of the first and second substances,

and the recovery process searches a target second class object with the recoverable capacity exceeding a preset recovery upper limit in the second storage pool in real time, rewrites each effective first class object which is not deleted in each first class object aggregated by the target second class object into the first storage pool, and deletes the aggregation mapping relation between each effective first class object and the target second class object in the first storage pool and the target second class object in the second storage pool.

7. The object storage platform of any one of claims 1-5, further comprising a pool of index records for recording index classifications for the first class of objects.

8. An object aggregation method applied to the object storage platform according to any one of claims 1 to 7, comprising:

aggregating the first class objects which are not aggregated to the second class objects into the second class objects in the second storage pool, and replacing the currently aggregated first class objects in the first storage pool with an aggregated mapping relation between the first class objects and the aggregated second class objects;

9. The method of claim 8, wherein periodically searching the first pool for objects of the first class that are not aggregated to objects of the second class comprises:

periodically playing back each write log and modification log which are positioned behind the aggregation checkpoint log in the log recording pool;

and taking the first-class objects oriented by the playback logs as the first-class objects which are not aggregated to the second-class objects.

10. The method of claim 9, further comprising, after aggregating the first class of objects that are not aggregated into the second class of objects in the second storage pool:

and according to the aggregation state, re-marking the aggregation check point log in each writing log and modification log in the log record pool.

11. The method of claim 8, wherein aggregating objects of a first class that are not aggregated into objects of a second class in a second storage pool comprises:

for each first class object which is not aggregated to a second class object in the first storage pool, if the aggregation capacity of the current second class object to the first class object in the second storage pool is smaller than the aggregation upper limit, directly adding the first class object in the current second class object; otherwise, taking the next second class object in the second storage pool as a new current second class object, and adding the first class object in the new current second class object.

12. The method of claim 8, further comprising:

detecting the recoverable capacity of each second class object in the second storage pool in real time, rewriting each valid first class object aggregated by a target second class object of which the recoverable capacity exceeds a preset recovery upper limit into the first storage pool, and deleting the aggregation mapping relation between each valid first class object in the first storage pool and the target second class object in the second storage pool.

13. The method of claim 12, wherein rewriting the respective valid first class objects aggregated by the target second class objects with the reclaimable capacity exceeding a preset reclamation upper limit into the first storage pool comprises:

searching an aggregation mapping relation between each first class object aggregated by the target second class object and the target second class object in the first storage pool respectively;

and rewriting the first class object with the non-empty aggregation mapping relation into the first storage pool as the effective first class object aggregated by the target second class object.

14. The method of claim 8, further comprising:

and if an operation request of the first class of objects is received, correspondingly updating the object information pointed by the operation request under the first storage pool and the second storage pool.

15. The method of claim 14, wherein if an operation request for an object of a first type is received, correspondingly updating object information pointed to by the operation request under the first storage pool and the second storage pool comprises:

and if the writing request of the first class object is received, directly storing the written first class object as the first class object which is not aggregated to the second class object into the first storage pool.

16. The method according to claim 15, after directly storing the first class objects written this time as the first class objects unaggregated to the second class objects in the first storage pool, further comprising:

and recording the write log of the first class object written at this time in the log record pool.

17. The method of claim 14, wherein if an operation request for an object of a first type is received, correspondingly updating object information pointed to by the operation request under the first storage pool and the second storage pool comprises:

if a reading request of the first class object is received, reading the content data of the first class object read at this time in the first storage pool;

and if the content data read in the first storage pool is empty, finding out an aggregation mapping relation between the first class objects read this time and the aggregated second class objects in the first storage pool, and continuously reading the content data of the first class objects read this time in the second class objects aggregated by the first class objects read this time in the second storage pool according to the aggregation mapping relation.

18. The method of claim 14, wherein if an operation request for an object of a first type is received, updating the object information pointed to by the operation request under the first storage pool and the second storage pool correspondingly comprises:

if a deleting request of the first class object is received and the deleted first class object is a first class object which is not aggregated to a second class object, deleting the deleted first class object in the first storage pool and deleting a write log of the deleted first class object in a log recording pool;

and if a deletion request of the first class objects is received and the deleted first class objects are aggregated to the second class objects, deleting the aggregation mapping relation between the deleted first class objects and the aggregated second class objects in the first storage pool, and updating the recoverable capacity of the aggregated second class objects of the deleted first class objects in the second storage pool.

19. The method of claim 14, wherein if an operation request for an object of a first type is received, correspondingly updating object information pointed to by the operation request under the first storage pool and the second storage pool comprises:

if a modification request of the first class object is received and the modified first class object is not aggregated to a second class object, updating the content data of the modified first class object in the first storage pool;

if a modification request of the first class object is received and the modified first class object is a first class object aggregated to a second class object, the modified first class object is directly written into the first storage pool, and when the modified first class object is aggregated to the second class object, the aggregation mapping relation between the first class object before modification and the aggregated second class object is deleted in the first storage pool.

20. The method according to claim 19, further comprising, after updating the content data of the currently modified first class object in the first storage pool, the following steps:

and recording the modification log of the first class object modified at this time in a log recording pool, and deleting the write log of the first class object modified at this time.

21. An object aggregation apparatus provided in the object storage platform according to any one of claims 1 to 7, comprising:

the object aggregation module is used for aggregating a first class of objects which are not aggregated to a second class of objects into the second class of objects in a second storage pool, and replacing the aggregated first class of objects in the first storage pool with an aggregated mapping relation between the first class of objects and the aggregated second class of objects;

22. A server, characterized in that the server comprises:

one or more processors;

storage means for storing one or more programs;

the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the object aggregation method of any of claims 8-20.

23. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the object aggregation method according to any one of claims 8 to 20.