CN115185951A

CN115185951A - Object uploading method, system, equipment and storage medium of storage system

Info

Publication number: CN115185951A
Application number: CN202210905689.7A
Authority: CN
Inventors: 赵煜; 王铂; 李欢欢; 陶桐桐
Original assignee: Jinan Inspur Data Technology Co Ltd
Current assignee: Jinan Inspur Data Technology Co Ltd
Priority date: 2022-07-29
Filing date: 2022-07-29
Publication date: 2022-10-14

Abstract

The application discloses an object uploading method, system, equipment and storage medium of a storage system, which are applied to the technical field of storage. The method comprises the following steps: setting a multilevel index directory of a bucket for the bucket in the storage system, and setting a virtual bucket fragment corresponding to the bucket fragment for each bucket fragment under the structure of the directory of each level; determining a target bucket, and determining a directory structure of an object to be uploaded under the target bucket based on the object name of the object to be uploaded; converting a directory structure of an object to be uploaded into a check value and mapping the check value to a preset value space to obtain a first value corresponding to the check value; and determining the serial number of the virtual barrel fragment which is closest to the first numerical value in the numerical space under the directory structure of the object to be uploaded, and writing the object information of the object to be uploaded into the barrel fragment corresponding to the virtual barrel fragment. By applying the scheme, the specification of the object stored in the single barrel is effectively improved, and the time consumption for orderly listing is guaranteed to be low. And the balance of storage is ensured.

Description

Object uploading method, system, equipment and storage medium of storage system

Technical Field

The present invention relates to the field of storage technologies, and in particular, to a method, a system, a device, and a storage medium for uploading an object in a storage system.

Background

Under the existing object storage framework, in order to guarantee the reliability and performance of storage, the specification of a single barrel for storing objects is limited to 1 hundred million. The specific reason is that, in an index slice of one bucket, a maximum of 10 ten thousand pieces of object information are recorded, and when the threshold is exceeded, a case of blocking may occur during cluster reconstruction. In the current scheme, object index information is recorded on a bucket slice in an object-map form, and the number of object-map pieces recorded by each bucket slice is limited to 10 ten thousand, so as to avoid stability and performance problems caused by exceeding. While a bucket defaults to 1024 index shards, if too many index shards are set, the time consumption for performing ordered enumeration increases because the objects between the index shards are unordered.

With the increasing of user data, in the current scheme, a plurality of buckets need to be created for data management, access and maintenance, which increases the difficulty of development and operation and maintenance, and therefore, for the specification of a single bucket for storing objects, the requirement of the single bucket for storing billions of objects is provided, but the time consumption for ensuring ordered enumeration is low.

In summary, how to effectively improve the specification of the objects stored in the single bucket and ensure the ordered enumeration is low in time consumption is a technical problem which needs to be solved by the technical personnel in the field at present.

Disclosure of Invention

The invention aims to provide an object uploading method, system, equipment and storage medium of a storage system, so as to effectively improve the specification of a single barrel for storing objects and ensure that the time consumption for ordered enumeration is low.

In order to solve the technical problems, the invention provides the following technical scheme:

an object uploading method of a storage system comprises the following steps:

setting a multi-level index directory of a bucket in a storage system, and setting a virtual bucket fragment corresponding to the bucket fragment for each bucket fragment under each level of directory structure;

determining a target bucket for storing object information of an object to be uploaded;

determining a directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded;

converting the directory structure of the object to be uploaded into a check value, and mapping the check value into a preset value space according to a preset mapping rule to obtain a first value corresponding to the check value;

determining the number of the virtual barrel fragment which is closest to the first numerical value in the numerical space under the directory structure of the object to be uploaded, and taking the barrel fragment corresponding to the virtual barrel fragment as a target barrel fragment;

writing the object information of the object to be uploaded into the target barrel fragment;

a plurality of barrel fragments are arranged under the directory structure of the object to be uploaded, and each barrel fragment is provided with a corresponding virtual barrel fragment; and the numbers of different virtual barrel fragments under the directory structure of the object to be uploaded are different, and the numbers of the virtual barrel fragments are mapped into the numerical space according to the mapping rule.

Preferably, the setting of the multi-level index directory of the bucket for the bucket in the storage system and the setting of the virtual bucket shard corresponding to the bucket shard for each bucket shard under each level of directory structure includes:

setting a primary index directory of a bucket in a storage system; the first-level index directory comprises m first-level directories and 1 metadata fragment, and the metadata fragment comprises k set barrel fragments;

setting a secondary index directory of a bucket in a storage system; aiming at each one-level directory in m one-level directories, the two-level index directory comprises n two-level directories and 1 metadata fragment corresponding to the one-level directory, and the metadata fragment comprises k barrel fragments;

setting a three-level index directory of a bucket in a storage system; for each secondary directory, the tertiary index directory comprises p tertiary directories and 1 metadata fragment corresponding to the secondary directory, and the metadata fragment comprises k set barrel fragments;

aiming at any 1 tertiary directory, the tertiary directory comprises k set barrel fragments;

setting q virtual barrel fragments corresponding to the barrel fragments aiming at each barrel fragment under each level of directory structure; m, n, p, k and q are positive integers.

Preferably, the method further comprises the following steps:

receiving a directory structure adjustment instruction, and adjusting the index directory level of the bucket.

Preferably, after determining the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded, the method further includes:

and when the directory structure of the object to be uploaded does not exceed the index directory level of the bucket and does not exist in the index directory of the bucket, adding the directory structure of the object to be uploaded to the index directory.

and when the directory structure of the object to be uploaded exceeds the index directory level of the bucket, outputting prompt information indicating that the object name of the object to be uploaded is wrong.

Preferably, the converting the directory structure of the object to be uploaded into a check value, and mapping the check value into a preset value space according to a preset mapping rule to obtain a first value corresponding to the check value includes:

and converting the directory structure of the object to be uploaded into a hash value, dividing the hash value by using a first fixed value, mapping the obtained remainder as the hash value into a preset numerical value space, and obtaining a first numerical value corresponding to the hash value.

Preferably, the method further comprises the following steps:

when the number of object information stored on any barrel fragment reaches a preset first threshold value, splitting the barrel fragment into 2 barrel fragments, and dividing a virtual barrel fragment corresponding to the barrel fragment before splitting into 2 parts, so that one part of the virtual barrel fragments corresponds to one barrel fragment after splitting, and the other part of the virtual barrel fragments corresponds to the other barrel fragment after splitting;

when the virtual barrel fragments corresponding to the barrel fragments before splitting are divided into 2 parts, the difference of the number of the virtual barrel fragments of the two parts is not more than 1.

An object upload system of a storage system, comprising:

the multi-level index directory building module is used for setting a multi-level index directory of the bucket for the bucket in the storage system and setting virtual bucket fragments corresponding to the bucket fragments for the bucket fragments under the directory structures at all levels;

the target bucket determining module is used for determining a target bucket for storing the object information of the object to be uploaded;

the to-be-uploaded object directory structure determining module is used for determining the directory structure of the to-be-uploaded object under the target bucket based on the object name of the to-be-uploaded object;

the numerical value mapping module is used for converting the directory structure of the object to be uploaded into a check value, and mapping the check value into a preset numerical value space according to a preset mapping rule to obtain a first numerical value corresponding to the check value;

a target barrel fragment determining module, configured to determine, in the directory structure of the object to be uploaded, a number of a virtual barrel fragment closest to the first value in the value space, and use a barrel fragment corresponding to the virtual barrel fragment as a target barrel fragment;

the writing module is used for writing the object information of the object to be uploaded into the target barrel fragment;

An object upload device of a storage system, comprising:

a memory for storing a computer program;

a processor for executing the computer program to implement the steps of the object uploading method of the storage system as described above.

A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the object upload method of a storage system as described above.

By applying the technical scheme provided by the embodiment of the invention, the multilevel index directory of the bucket is set for the bucket in the storage system, when the object is required to be uploaded subsequently, the target bucket for storing the object information of the object to be uploaded is determined, and then the directory structure of the object to be uploaded under the target bucket is determined based on the object name of the object to be uploaded, namely, different objects to be uploaded with similar object names can be placed under the same or similar directory structure. In other words, in the scheme of the application, based on the object name of the object to be uploaded, the multi-level index directory of the bucket is utilized, so that different object information can be divided preliminarily, and time consumption is short if the bucket of the application is utilized for orderly listing. Moreover, the scheme of the application can not limit the quantity of the barrel fragments, namely the specification of the single barrel storage object can reach the billion specifications required at present or even be higher.

In addition, after the directory structure of the object to be uploaded under the target bucket is determined, the directory structure of the object to be uploaded can be converted into a check value, the check value is mapped to a preset value space according to a preset mapping rule, and a first value corresponding to the check value is obtained. Then, determining the number of the virtual barrel fragment which is closest to the first numerical value in the numerical value space under the directory structure of the object to be uploaded, taking the barrel fragment corresponding to the virtual barrel fragment as a target barrel fragment, and writing the object information of the object to be uploaded into the target barrel fragment; load balancing is facilitated by the fact that for a plurality of bucket fragments under the same directory structure, objects to be uploaded placed in the directory structure can be evenly distributed to the bucket fragments under the directory structure.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a flowchart illustrating an implementation of an object uploading method of a storage system according to the present invention;

FIG. 2 is a block diagram of a 3-level index directory according to an embodiment of the present invention;

fig. 3 is a schematic diagram illustrating a correspondence relationship between a bucket fragment and a virtual bucket fragment according to an embodiment of the present invention;

fig. 4 is a schematic diagram illustrating mapping of virtual bucket fragment numbers and check values to a preset numerical space according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of an object uploading system of a storage system in the present invention.

Detailed Description

The core of the invention is to provide an object uploading method of a storage system, which effectively improves the specification of storing objects in a single barrel and ensures that the time consumption of ordered enumeration is low. And the storage balance is ensured, namely, the storage balance of all the barrel fragments under the same directory structure is facilitated.

In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It should be apparent that the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.

Referring to fig. 1, fig. 1 is a flowchart illustrating an implementation of an object uploading method of a storage system according to the present invention, where the object uploading method of the storage system may include the following steps:

step S101: a multi-level index directory of the bucket is set for the bucket in the storage system, and a virtual bucket fragment corresponding to the bucket fragment is set for each bucket fragment under the directory structure of each level.

In particular, the storage system of the present application may generally be a distributed object storage system. In a storage system, one or more buckets may be provided, each bucket may implement aspects of the present application. Certainly, after the scheme of the application is considered, a single barrel can store billions of pieces of object information, so that only a single barrel needs to be arranged in some occasions, and the difficulty of development, operation and maintenance of workers is reduced.

According to the scheme, the multi-level index directory of the bucket needs to be set, when the object information of the object to be uploaded is stored subsequently, the directory structure of the object to be uploaded is determined according to the name of the object to be uploaded, and therefore different objects to be uploaded with similar object names can be placed under the same or similar directory structures. That is, based on the object name of the object to be uploaded, the object information can be preliminarily divided by using the multilevel index directory of the bucket, so that time consumption is short if the bucket of the application is used for orderly enumeration.

This application still need set up barrel piece in multistage index directory, and the barrel piece quantity that specifically sets up can be set for and adjust as required, for example under each directory of minimum one-level, all sets for the barrel piece of certain quantity. For each barrel fragment under each level of directory structure, the application also needs to set a virtual barrel fragment corresponding to the barrel fragment for each barrel fragment under each level of directory structure, and the set virtual barrel fragment can be used for realizing the balance of object information storage in subsequent steps.

There are various specific ways to set the multi-level index directory of the bucket. For example, in a specific embodiment of the present invention, step S101 may specifically include:

setting a first-level index directory of a bucket for the bucket in the storage system; the first-level index directory comprises m first-level directories and 1 metadata fragment, and the metadata fragment comprises k set barrel fragments;

setting a secondary index directory of the bucket for the bucket in the storage system; aiming at each primary directory in the m primary directories, the secondary index directory comprises n secondary directories and 1 metadata fragment corresponding to the primary directory, and the metadata fragment comprises k set barrel fragments;

setting a three-level index directory of buckets for the buckets in the storage system; aiming at each secondary directory, the tertiary index directory comprises p tertiary directories and 1 metadata fragment corresponding to the secondary directory, and the metadata fragment comprises k set barrel fragments;

setting q virtual barrel fragments corresponding to the barrel fragments aiming at each barrel fragment under each level of directory structure; and m, n, p, k and q are all positive integers.

In this embodiment, a 3-level index directory is provided for the bucket, where m level directories and 1 piece of metadata are provided under the level index directory. And m is a positive integer, and the specific value can be selected according to the requirement. Setting 1 metadata fragment under the first-level index directory, considering that in some occasions, object information of an object to be uploaded is allowed to be directly stored under the first-level index directory, so that 1 metadata fragment is set, and the metadata fragment comprises k set barrel fragments. The metadata fragment shown in fig. 2 is represented as a "file", i.e. a metadata fragment composed of objects without directory structure in the bucket, which may be named, for example, bucketID + file.

In addition, for the primary index directory of the bucket, a primary index table may be used for representation, for example, table one is the primary index table in a specific case.

Table one:

first level directory 1	buckettID + Primary directory 1
		First level directory 2	bucketID + Primary directory 2
...	...
		First level directory m	bucketID + Primary directory m
Document	bucketID+file

The left side in table one is a key, which is the first-level directory structure reflected by the first-level index table, i.e., the number of the first-level directory and 1 metadata fragment, and the right side is a value, which is the name of the corresponding two-level index table. For example, table one may be named bucketID + index.

For each of the m primary directories, n secondary directories and 1 metadata fragment corresponding to the primary directory are set under the secondary index directory. n is a positive integer, and the specific value can be selected according to the requirement.

In the secondary index directory under any 1 primary directory, 1 metadata fragment is set, in consideration of some occasions, object information of an object to be uploaded is allowed to be directly stored under the secondary index directory, and therefore, for each primary directory in m primary directories, 1 metadata fragment corresponding to the primary directory is set under the secondary index directory. The metadata fragment shown in fig. 2 is represented as "primary directory 1/file", i.e. a metadata fragment composed of objects without directory structure under the secondary index directory, and the metadata fragment may be named as: bucketID + Primary directory number + file. The primary directory number refers to the number of the primary directory corresponding to the metadata fragment, for example, the metadata fragment under the secondary index directory shown in fig. 2 may be named: bucketID + first level directory 1+ file.

Since there are m primary directories, the secondary index directory of the bucket can be represented by m secondary index tables, for example, table two is one of the secondary index tables.

A second table:

primary directory 1/secondary directory 1	bucketID + Primary directory 1+ Secondary directory 1
		Primary directory 1/secondary directory 2	bucketID + Primary directory 1+ Secondary directory 2
...	...
		Primary directory 1/secondary directory n	bucketID + Primary directory 1+ Secondary directory n
First level directory 1/file	buckettID + first-level directory 1+ file

The left side in table two is the key of the secondary index table, that is, the secondary directory structure reflected by the secondary index table, that is, each secondary directory number under the

primary directory

1, and 1 metadata fragment. The value is on the right and the name of the corresponding tripolar index table.

Because m primary directories are arranged and n secondary directories are arranged under each primary directory, m × n secondary directories exist. For each secondary directory, p tertiary directories and 1 metadata fragment corresponding to the secondary directory are set in the tertiary index directory. And p is a positive integer, and specific values can be selected according to needs.

In a tertiary index directory under any 1 secondary directory, 1 metadata fragment is set, and the metadata fragment comprises k set barrel fragments, which allows object information of an object to be uploaded to be directly stored under the tertiary index directory in consideration of some occasions, so that 1 metadata fragment corresponding to the secondary directory is set under the tertiary index directory for each secondary directory. The metadata fragment shown in fig. 2 is represented as "primary directory 1/secondary directory 3/file", i.e. a metadata fragment composed of objects without directory structure under the tertiary index directory, which may be named: bucketID + Primary directory number + Secondary directory number + File. The primary directory number and the secondary directory number refer to the number of the primary directory and the number of the secondary directory corresponding to the metadata fragment, for example, the metadata fragment under the tertiary index directory shown in fig. 2 may be named as: buckettID + primary directory 1+ secondary directory 3+ file.

For the bucket's three-level index directory, it can be represented by a plurality of three-level index tables, for example, table three shows one of the three-level index tables.

Table three:

the left side in table three is the key of the tertiary index table, that is, the tertiary directory structure reflected by the tertiary index table, that is, each tertiary directory number and 1 metadata fragment under the primary directory 1/secondary directory 3. The right side is a value, which may be the name of the corresponding table of bucket shards under the corresponding three-level directory structure. Table three may be named buckettID + Primary directory 1+ Secondary directory 3.

For any 1 tertiary directory, the tertiary directory includes k set bucket fragments, for example, k bucket fragments under "primary directory 1/secondary directory 3/tertiary directory 2" are shown in fig. 2. k is a positive integer, and the specific value can be selected according to the requirement.

Setting q virtual barrel fragments corresponding to the barrel fragments aiming at each barrel fragment under each level of directory structure; q is a positive integer, and the specific value can be selected according to the requirement. In fig. 2, it is indicated that bucket shard 2 of the k bucket shards under "primary directory 1/secondary directory 3/tertiary directory 2" corresponds to q virtual bucket shards.

Step S102: and determining a target bucket for storing the object information of the object to be uploaded.

Specifically, there are various ways to determine the target bucket for storing the object information of the object to be uploaded, for example, the determination may be performed based on the object name of the uploaded object, or, for example, the determination may be performed by an upload instruction, and the like. Of course, in some cases, when only 1 bucket is set, the bucket may be directly used as a target bucket for storing object information of an object to be uploaded.

Step S103: and determining a directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded.

The object name of the object to be uploaded can comprise a plurality of items of information, and the directory structure of the object to be uploaded under the target bucket is determined based on the object name of the object to be uploaded. For example, in one case, the beginning of the object name of the object to be uploaded is bucketID + primary directory 1, which indicates that the object information of the object to be uploaded needs to be stored in a bucket slice in "primary directory 1/file" in fig. 2. For another example, the beginning of the object name of the object to be uploaded is buckettid + primary directory 1+ secondary directory 3+ tertiary directory 2, which illustrates that the object information of the object to be uploaded needs to be stored in a bucket slice in "primary directory 1/secondary directory 3/tertiary directory 2" in fig. 2 subsequently.

Step S104: and converting the directory structure of the object to be uploaded into a check value, and mapping the check value into a preset value space according to a preset mapping rule to obtain a first value corresponding to the check value.

When the directory structure of the object to be uploaded is converted into the check value, there are various specific algorithms, for example, considering that the hash value is a commonly used check value, the directory structure of the object to be uploaded can be converted into the hash value.

There are also various ways of mapping the check value into the preset value space, for example, one convenient way is to make division residue.

That is, in an embodiment of the present invention, step S104 may specifically include:

The hash value is divided by using the first fixed value, that is, the first fixed value is used as a divisor, the hash value is used as a dividend, and the obtained remainder is used as a hash value and is mapped to a first numerical value corresponding to the hash value, which is obtained in a preset numerical value space.

For example, in one case, the numerical space is 0 to 2 ³¹ -1. The first fixed value is 2 ³¹ -1, after mapping, obtaining a first value corresponding to the check value falling between 0 and 2 ³¹ -1.

Step S105: and determining the number of the virtual barrel fragment closest to the first numerical value in the numerical space under the directory structure of the object to be uploaded, and taking the barrel fragment corresponding to the virtual barrel fragment as a target barrel fragment.

A plurality of barrel fragments are arranged under the directory structure of the object to be uploaded, and each barrel fragment is provided with a corresponding virtual barrel fragment. For example, in the above example, if the directory structure of the object to be uploaded is first level directory 1/second level directory 3/third level directory 2, k bucket fragments are arranged under the directory structure. Each of the k bucket slices has a respective corresponding virtual bucket slice. For example, k =128, each bucket shard corresponds to 100 virtual bucket shards, i.e., p is 100, i.e., under the directory structure of the object to be uploaded, there are 128 × 100=12800 virtual bucket shards in total.

For example, in fig. 3, bucket shard 0 corresponds to virtual bucket shard 0.0 through virtual bucket shard 0.99. Bucket fragment 1 corresponds to virtual bucket fragment 1.0 through virtual bucket fragment 1.99. Bucket shard 127 corresponds to virtual bucket shard 127.0 through virtual bucket shard 127.99.

And the numbers of different virtual barrel fragments under the directory structure of the object to be uploaded are different, and the numbers of all the virtual barrel fragments are mapped into the numerical space according to the mapping rule. That is to say, in the above example, the numbers of the 12800 virtual bucket fragments are different from each other, and the numbers of the 12800 virtual bucket fragments are all mapped into the value space according to the mapping rule, see fig. 4, where the 12800 virtual bucket fragments are all mapped into the value spaces 0 to 2 according to the mapping rule ³¹ 1 is mentioned above.

For example, the fourth table is a corresponding table of bucket fragments in a certain three-level directory structure, that is, it represents the corresponding relationship between each bucket fragment in the directory structure and a virtual bucket fragment.

Table four:

hash value (virtual barrel fragment 0# 0)	Barrel fragment 0
		Hash value (virtual bucket fragmentation)0#n)	Barrel fragment 0
Hash value (virtual barrel fragment 0# -99)	Barrel fragment 0
		Hash value (virtual barrel fragment 1# 0)	Barrel fragment 1
Hash value (virtual barrel fragment 1 #)	Barrel slicing 1
		Hash value (virtual barrel fragment 1# -99)	Barrel fragment 1
...	...
		Hash value (virtual barrel fragment 127# 0)	Barrel fragment 127
Hash value (virtual barrel fragment 127# n)	Barrel fragment 127
		Hash value (virtual barrel fragment 127# 99)	Barrel fragment 127

It can be appreciated that the location is fixed after 12800 virtual bucket shards are mapped to the value space. Therefore, in practical applications, for convenience of calculation, the number and the number of the set virtual bucket shards are consistent under different directory structures. That is, for example, for each different directory, such as the primary directory 1/secondary directory 3/tertiary directory 1, the primary directory 1/secondary directory 5/tertiary directory 7, k =128 may be set, each bucket fragment corresponds to 100 virtual bucket fragments, and the virtual bucket fragments start from 0.0 and are numbered with a step size of 0.01 and a value of 127.99.

Obj1 to obj6 in fig. 4 indicate the first values of the 6 objects to be uploaded in the value space, and it can be seen that for obj1, the number of the virtual bucket slice closest to obj1 in the value space is virtual slice 0.0. For obj2, the number of the virtual bucket slice closest to obj2 in the value space is virtual slice 1.0. For obj3, the number of the virtual bucket slice closest to obj3 in the value space is virtual slice 127.0. For obj4, the number of the virtual bucket slice closest to obj4 in the value space is virtual slice 1.99.

Step S106: writing object information of an object to be uploaded into the target barrel fragment;

after the target barrel fragment is determined, the object information of the object to be uploaded can be written into the target barrel fragment. For example, in the above example, the directory structure of the object to be uploaded is first level directory 1/second level directory 3/third level directory 2, and after the directory structure of the object to be uploaded is converted into the check value, the directory structure is mapped into the preset value space to obtain the first value corresponding to the check value, for example, the number of the virtual bucket fragment closest to the first value in the value space is 126.58, and the bucket fragment corresponding to the virtual bucket fragment is 126, so that the object information of the object to be uploaded can be written into the bucket fragment 126 under the first level directory 1/second level directory 3/third level directory 2.

In an embodiment of the present invention, the method may further include:

and receiving a directory structure adjusting instruction, and adjusting the index directory level of the bucket.

In the embodiment of fig. 2, a 3-level index directory is provided, and in this embodiment, directory structure adjustment is allowed, i.e., config may be adjusted to modify the index directory level of the bucket after receiving a directory structure adjustment instruction.

In a specific embodiment of the present invention, after determining a directory structure of an object to be uploaded under a target bucket based on an object name of the object to be uploaded, the method may further include:

For example, in one embodiment, the index directory of the bucket is ranked 3, and 100 first-level directories are provided, i.e., m is 100. For example, based on the object name of the object to be uploaded, the directory structure of the object to be uploaded under the target bucket is determined to be a primary directory 104, a secondary directory 3 or a tertiary directory 2. It can be seen that this directory structure does not exist in the index directory of the bucket, i.e., the key of primary directory 104 does not exist in the primary index table. However, the index directory level of the directory structure is 3 levels, and the directory structure of the object to be uploaded can be added to the index directory without exceeding the index directory level of the bucket. Namely, the corresponding first-level index table, second-level index table and third-level index table are adjusted, so that a directory structure of 'first-level directory 104/second-level directory 3/third-level directory 2' is added to the index directory of the bucket.

It can be seen that, in this embodiment, the adjustment of the index directory of the bucket is allowed according to the object name of the object to be uploaded, and the flexibility is high. In practical applications, the first-level index table, each second-level index table, and each third-level index table may be stored in the cache.

In a specific embodiment of the present invention, after determining a directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded, the method may further include:

and when the directory structure of the object to be uploaded exceeds the index directory level of the bucket, outputting prompt information indicating that the name of the object to be uploaded is wrong.

Still taking the index directory level of the bucket as 3 levels, and setting 100 first-level directories in the bucket as an example, for example, the determined directory structure of the object to be uploaded under the target bucket is first-level directory 104/second-level directory 3/third-level directory 2/fourth-level directory 1. It can be seen that the directory structure does not exist in the index directory of the bucket, and the index directory level of the directory structure is 4 levels, which has exceeded the index directory level of the bucket, which is usually caused by an object name error of an object to be uploaded, and therefore, prompt information indicating that the object name of the object to be uploaded is erroneous can be output.

In one embodiment of the present invention, the method further comprises:

When the number of the object information stored on any barrel fragment reaches the preset first threshold, it indicates that the number of the object information stored on the barrel fragment is large, and the object information stored on the barrel fragment continues, which may cause problems of stability and performance. Therefore, the original bucket fragment is split to obtain 2 new bucket fragments, and the virtual bucket fragment corresponding to the bucket fragment before splitting is divided into 2 parts, usually 2 parts with equal number. Of course, when the number of virtual barrel slices corresponding to the barrel slice before splitting is an odd number, the virtual barrel slices cannot be exactly divided into 2 parts with equal number, and the virtual barrel slices can be divided into 2 parts with the number difference not exceeding 1.

For example, original bucket shard 127 corresponds to virtual bucket shards 127.0-127.99, then after splitting, bucket shard 127 and bucket shard 128 are obtained, new bucket shard 127 corresponds to virtual bucket shards 127.0-127.49, for example, and new bucket shard 128 corresponds to virtual bucket shards 127.50-127.99.

It can be seen that, when the embodiment splits the bucket fragments, the corresponding relationship between the virtual bucket fragments needs to be adjusted, but the process does not affect other bucket fragments, so that errors are not easy to occur, and even if errors occur, the influence is small, that is, only a single bucket fragment is affected. In addition, the scheme is also beneficial to guaranteeing the normal operation of the service, namely when a certain bucket is split, the operation of the service based on other buckets is not influenced.

By applying the technical scheme provided by the embodiment of the invention, the multilevel index directory of the bucket is set for the bucket in the storage system, when the object is required to be uploaded subsequently, the target bucket for storing the object information of the object to be uploaded is determined, and then the directory structure of the object to be uploaded under the target bucket is determined based on the object name of the object to be uploaded, namely, different objects to be uploaded with similar object names can be placed under the same or similar directory structure. In other words, in the scheme of the application, based on the object name of the object to be uploaded, the multi-level index directory of the bucket is utilized, so that different object information can be divided preliminarily, and time consumption can be short if the bucket of the application is utilized for orderly enumeration. In addition, the number of barrel fragments cannot be limited by the scheme, namely the specification of a single barrel storage object can reach the billion specification required at present and is even higher.

In addition, after the directory structure of the object to be uploaded under the target bucket is determined, the directory structure of the object to be uploaded can be converted into a check value, the check value is mapped to a preset value space according to a preset mapping rule, and a first value corresponding to the check value is obtained. Then determining the number of the virtual barrel fragment closest to the first numerical value in the numerical value space under the directory structure of the object to be uploaded, taking the barrel fragment corresponding to the virtual barrel fragment as a target barrel fragment, and writing the object information of the object to be uploaded into the target barrel fragment; load balancing is facilitated by the fact that for a plurality of bucket fragments under the same directory structure, objects to be uploaded placed in the directory structure can be evenly distributed to the bucket fragments under the directory structure.

Corresponding to the above method embodiment, the embodiment of the present invention further provides an object uploading system of a storage system, which can be referred to in correspondence with the above.

Referring to fig. 5, a schematic structural diagram of an object uploading system of a storage system in the present invention is shown, including:

a multi-level index directory constructing module 501, configured to set a multi-level index directory of a bucket for the bucket in the storage system, and set a virtual bucket fragment corresponding to the bucket fragment for each bucket fragment under each level directory structure;

a target bucket determining module 502, configured to determine a target bucket used for storing object information of an object to be uploaded;

the to-be-uploaded object directory structure determining module 503 is configured to determine, based on the object name of the to-be-uploaded object, a directory structure of the to-be-uploaded object under the target bucket;

the numerical value mapping module 504 is configured to convert a directory structure of an object to be uploaded into a check value, and map the check value into a preset numerical value space according to a preset mapping rule, so as to obtain a first numerical value corresponding to the check value;

a target bucket fragment determining module 505, configured to determine a number of a virtual bucket fragment closest to the first numerical value in the numerical space under the directory structure of the object to be uploaded, and use a bucket fragment corresponding to the virtual bucket fragment as the target bucket fragment;

a writing module 506, configured to write object information of an object to be uploaded into the target bucket fragment;

the method comprises the steps that a plurality of barrel fragments are arranged under a directory structure of an object to be uploaded, and each barrel fragment is provided with a corresponding virtual barrel fragment; the numbers of different virtual barrel fragments under the directory structure of the object to be uploaded are different, and the numbers of the virtual barrel fragments are mapped to a numerical value space according to a mapping rule.

In an embodiment of the present invention, the multi-level index directory constructing module 501 is specifically configured to:

setting a first-level index directory of the buckets for the buckets in the storage system; the first-level index directory comprises m first-level directories and 1 metadata fragment, and the metadata fragment comprises k set barrel fragments;

aiming at any 1 tertiary catalog, the tertiary catalog comprises k set barrel fragments;

In an embodiment of the present invention, the multi-level index directory building module 501 is further configured to:

In a specific embodiment of the present invention, after the to-be-uploaded object directory structure determining module 503 determines, based on the object name of the to-be-uploaded object, the directory structure of the to-be-uploaded object in the target bucket, the multi-level index directory constructing module 501 is further configured to:

In a specific embodiment of the present invention, after the to-be-uploaded object directory structure determining module 503 determines, based on the object name of the to-be-uploaded object, a directory structure of the to-be-uploaded object under the target bucket, the method further includes:

and the error reporting module is used for outputting prompt information indicating that the object name of the object to be uploaded is wrong when the directory structure of the object to be uploaded exceeds the index directory level of the bucket.

In an embodiment of the present invention, the value mapping module 504 is specifically configured to:

In an embodiment of the present invention, the system further includes a bucket classification control module, configured to:

when the virtual barrel fragments corresponding to the barrel fragments before splitting are divided into 2 parts, the number difference of the two parts of virtual barrel fragments is not more than 1.

Corresponding to the above method and system embodiments, the embodiments of the present invention further provide an object uploading device of a storage system and a computer readable storage medium, which may be referred to in correspondence with the above.

The object uploading device of the storage system may include:

a memory for storing a computer program;

a processor for executing a computer program to implement the steps of the object uploading method of the storage system as in any of the above embodiments.

The computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the object uploading method of the storage system as in any of the above embodiments. A computer-readable storage medium as referred to herein may include Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising one of 8230; \8230;" 8230; "does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.

Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The principle and the embodiment of the present invention are explained by applying specific examples, and the above description of the embodiments is only used to help understanding the technical solution and the core idea of the present invention. It should be noted that, for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall into the protection scope of the present invention.

Claims

1. An object uploading method of a storage system, comprising:

converting the directory structure of the object to be uploaded into a check value, and mapping the check value into a preset numerical value space according to a preset mapping rule to obtain a first numerical value corresponding to the check value;

2. The object uploading method of the storage system according to claim 1, wherein the setting of the multi-level index directory of the bucket for the bucket in the storage system and the setting of the virtual bucket shard corresponding to the bucket shard for each bucket shard under each level of directory structure comprises:

3. The object uploading method of the storage system according to claim 2, further comprising:

4. The method for uploading objects in a storage system according to claim 1, wherein after determining a directory structure of the objects to be uploaded under the target bucket based on object names of the objects to be uploaded, the method further comprises:

5. The object uploading method of the storage system according to claim 4, wherein after determining the directory structure of the object to be uploaded under the target bucket based on the object name of the object to be uploaded, the method further comprises:

6. The object uploading method of the storage system according to claim 1, wherein converting a directory structure of the object to be uploaded into a check value, and mapping the check value into a preset value space according to a preset mapping rule to obtain a first value corresponding to the check value comprises:

7. The object uploading method of the storage system according to any one of claims 1 to 6, further comprising:

8. An object upload system of a storage system, comprising:

the multi-level index directory building module is used for setting a multi-level index directory of the bucket for the bucket in the storage system and setting virtual bucket fragments corresponding to the bucket fragments for each bucket fragment under each level of directory structure;

the to-be-uploaded object directory structure determining module is used for determining a directory structure of the to-be-uploaded object under the target bucket based on the object name of the to-be-uploaded object;

a plurality of barrel fragments are arranged under the directory structure of the object to be uploaded, and each barrel fragment is provided with a corresponding virtual barrel fragment; and the serial numbers of different virtual barrel fragments under the directory structure of the object to be uploaded are different, and the serial numbers of all the virtual barrel fragments are mapped into the numerical space according to the mapping rule.

9. An object upload apparatus of a storage system, comprising:

a memory for storing a computer program;

a processor for executing the computer program to implement the steps of the object upload method of the storage system according to any of claims 1 to 7.

10. A computer-readable storage medium, characterized in that a computer program is stored thereon, which computer program, when being executed by a processor, carries out the steps of an object uploading method of a storage system according to any of the claims 1 to 7.