CN115587390A - Method for realizing public cloud object storage - Google Patents
Method for realizing public cloud object storage Download PDFInfo
- Publication number
- CN115587390A CN115587390A CN202211587570.6A CN202211587570A CN115587390A CN 115587390 A CN115587390 A CN 115587390A CN 202211587570 A CN202211587570 A CN 202211587570A CN 115587390 A CN115587390 A CN 115587390A
- Authority
- CN
- China
- Prior art keywords
- object storage
- data
- bucket
- files
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 230000004044 response Effects 0.000 claims abstract description 7
- 238000007906 compression Methods 0.000 claims description 31
- 230000006835 compression Effects 0.000 claims description 31
- 238000013144 data compression Methods 0.000 claims description 25
- 238000007726 management method Methods 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 13
- 230000004048 modification Effects 0.000 claims description 5
- 238000012986 modification Methods 0.000 claims description 5
- 238000012856 packing Methods 0.000 claims description 4
- 238000002955 isolation Methods 0.000 abstract description 8
- 238000010586 diagram Methods 0.000 description 13
- 239000012634 fragment Substances 0.000 description 6
- 239000002699 waste material Substances 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for realizing public cloud object storage, which is applied to an object storage framework for deploying object storage service and an object storage management end, wherein the object storage service deploys a single-process example and comprises the following steps: responding to the opening of object storage, generating random characters by an object storage management end, generating a first rule based on a prefix of a random character string, calling an object storage service interface to issue to an object storage service, and establishing a second rule by the object storage service based on the first rule; in response to the creation of the data bucket, the object storage management terminal creates a data bucket name based on the random characters and issues the data bucket name to the object storage service; the object storage service judges whether to create a data bucket based on the second rule and the data bucket name; the method for realizing public cloud object storage solves the problems of excessive resource consumption of multiple tenants, barrel naming and data security isolation among different tenants, and efficiently realizes charging, storage and query of mass files.
Description
Technical Field
The invention relates to the field of object storage, in particular to a method for realizing public cloud object storage.
Background
The object storage is a hotspot of storage research in recent years, and compared with file storage, the object storage is simple to implement, the development period is shortened, and meanwhile, erasure codes are supported, so that the storage space is greatly reduced. However, the object storage software can be realized by taking the 'private cloud' and the 'public cloud' into consideration. The object storage system only needs one tenant in the private cloud scene; and in a public cloud scenario, many tenants need to be supported. The multi-tenant needs to consider the problem of the memory naming conflict of each tenant.
In some implementations of vendors with public clouds, storage namespaces of different tenants are public, that is, conflicts occur, so a tenant may find that creation cannot be performed when creating a storage space because other tenants have created a storage space of the name.
MinIO is object storage software that is heavily used in current private clouds, is simple to deploy, run, and provides sophisticated AWS S3 functionality. In the private cloud environment, only one tenant is needed, while in the public cloud environment, a large number of tenants are needed to provide services for different tenants, and the mode of providing multi-tenant services by adopting MinIO is not ideal.
In the prior art, as shown in fig. 1, in the multi-instance multi-tenant scheme provided by MinIO, the multi-instance multi-tenant scheme is implemented in a manner of creating a set of MinIO processes in a MinIO cluster for each different tenant, that is, each tenant bottom layer directly runs one MinIO process on each MinIO node, which is relatively simple and can quickly implement multi-tenant.
Each MinIO process consumes at least 200MB of memory, even if the tenant has no object, the creation also consumes resources, and the nodes become larger gradually in the using process, if 1000 tenants exist, each node only runs the MinIO process and needs to consume 200GB of memory, which is obviously not feasible. And each MinIO process needs to independently use one port for internal communication, which causes great waste of memory resources and port resources, and is inconvenient for multi-process management and high in management cost.
In addition, as the MinIO is mainly designed for the private cloud, the MinIO cannot directly acquire the flow and the request volume of each packet when the public cloud object storage is realized, so that charging cannot be performed according to the packets in a public cloud scene.
Meanwhile, in a public cloud scene, the query speed of the small files by adopting MinIO object storage is not ideal, and on one hand, the small files cause a plurality of fragments of the disk, so that the query speed is low, and the fragments cause the low utilization rate of the disk. On the other hand, a large number of users in the public cloud share one object storage, and each user may upload small files of any size and any number, which may cause the query speed to be slow, and may also affect the speed of other users accessing the object storage.
The prior art adopts MinIO to realize the storage of public cloud objects, has the following main defects,
1. for each tenant, all MinIO servers in the cluster need to start a MinIO process, which causes great waste of memory resources and port resources;
2. each tenant corresponds to a group of MinIO processes in the MinIO cluster, and when the number of tenants is large, the number of processes is too large to manage;
3. because MinIO cannot directly count the flow and the request amount of each bucket. Therefore, charging cannot be realized when MinIO realizes public cloud object storage;
4. under a public cloud scene, the query performance is low, the number of disk fragments is large, and the storage utilization rate is low due to the massive small files in the object storage;
5. the tenant may fail due to naming conflict when creating the bucket;
6. when a tenant has many sockets, a large number of rules need to be configured for socket access limitation.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides a method for realizing public cloud object storage, which solves the problems of excessive resource consumption caused by multiple tenants under a public cloud scene, bucket naming among different tenants, data security isolation among the multiple tenants, charging of the public cloud object storage and how to efficiently store and query mass files.
The invention provides a method for realizing public cloud object storage, which is used for deploying at least an object storage service and an object storage architecture of an object storage management end, wherein the object storage service deploys a single-process instance, and the method for realizing the public cloud object storage specifically comprises the following steps: responding to the opening of object storage, generating random characters by an object storage management end, generating a first rule based on a prefix of a random character string, calling an object storage service interface to issue to an object storage service, and establishing a second rule by the object storage service based on the first rule; in response to the creation of the data bucket, the object storage management terminal creates a data bucket name based on the random characters and issues the data bucket name to the object storage service; and the object storage service judges whether to create the data bucket based on the second rule and the data bucket name, if the data bucket name meets the second rule, the creation of the data bucket is successful, and if not, the creation of the data bucket fails.
The problem of resource waste of multiple instances is solved by deploying a single process, and tenants are distinguished by random character prefixes, so that the problem of bucket naming conflict of multiple tenants and resource isolation between users caused by a single instance is solved.
Further, the object storage service is further included to manage the data buckets based on a second rule.
Furthermore, the object storage architecture of the application further comprises a proxy server arranged between the object storage service and the object storage management terminal, wherein the proxy server deployment compression module is used for compressing files in the data bucket and/or the deployment statistics module is used for counting the access flow of the data bucket.
The data compression is directly realized by deploying the object storage service function module, and can also be realized by deploying the compression module in the proxy server.
Further, the object storage service also creates data compression buckets associated with the data buckets, or creates data compression buckets associated with the data buckets based on the create data bucket requests forwarded by the compression module.
Storing infrequently used small files in a data bucket by creating a data compression bucket associated with the data bucket
And the file is compressed, and efficient storage is realized.
Further, compressing the small file based on the data compression bucket specifically includes: scanning a data bucket based on a preset period, and acquiring file information in the data bucket; based on the comparison between the acquired file information and a preset compression threshold, packing and compressing the files meeting the preset compression condition, naming the compressed files according to a preset rule, and storing the files into a data compression barrel associated with the data barrel; and deleting the compressed files in the data bucket.
Further, the compressing the file meeting the preset compression condition based on the comparison between the acquired file information and the preset compression threshold specifically includes: obtaining the time of the file which is not changed based on the last modification time and the current time in the obtained file information; comparing the unchanged time with a preset time threshold, and if the unchanged time is greater than the preset time threshold, taking the file meeting the conditions as a first compressed file set; if the size of the files in the first compressed file set is smaller than a preset size threshold value, summarizing the files meeting the conditions to a second compressed file set; and comparing the number of the files in the second compressed file set with a preset number threshold, and if the number of the files in the second compressed file set is greater than or equal to the preset number threshold, taking the files in the second compressed file set as files meeting the preset compression condition.
Further, querying a file list based on the data compression bucket includes: responding to a file request for inquiring the data bucket, and inquiring the compressed files in the data compression bucket; reading a compressed file list of the compressed file based on a preset rule, and reading all file lists in the data bucket; a summary compressed file list and a list of files in the data bucket are returned.
And the file list in the compressed data bucket and the file list in the data bucket are integrated, so that the massive files can be efficiently inquired.
Further, the method also includes deploying a statistical module based on the proxy server so as to count the access flow of the data bucket, and specifically includes: a counting module creates a barrel data table used for storing data barrel information; the proxy server receives the message and forwards the message to the statistical module; the counting module acquires message information, searches whether a data bucket name in the message exists or not based on a bucket data table, and if not, creates corresponding data bucket information in the message in the bucket data table; storing the message length and the access amount of a corresponding data bucket in the message to a bucket data table; storing the information in the bucket data table to a file corresponding to the data bucket based on the current time and a preset period; and responding to the query condition, and formatting data corresponding to the query condition in the return file.
Further, the formatting the data corresponding to the query condition in the return file in response to the query condition specifically includes: based on the query conditions, the statistical module queries the files corresponding to the data bucket, reads the information corresponding to the query conditions in the files, obtains the timestamp, the flow of the preset period, the access amount and the bucket flow of the time period based on the information in the files, and returns the access amount and the bucket flow according to the preset format. Through the statistical module deployed in the proxy server, the high-efficiency charging of the public cloud object storage is achieved, and the traffic of each time period can be conveniently counted.
The invention has the beneficial effects that:
1. the multi-tenant is realized by using the mode of deploying the single instance by the object storage service, the problem of resource waste of the multi-instance is solved, one process is provided for the multi-tenant, and the problem of resource isolation between users brought by the single instance is solved.
2. And a data compression barrel of the associated data barrel is set for storing compressed files corresponding to the small files meeting the compression conditions, so that the problem of fragments caused by excessive files is solved.
3. When the data bucket file list is inquired, the compressed file is stored in a specific file, so that the file list can be directly obtained without decompressing all data.
4. The charging module is arranged in the reverse proxy server, so that the total access volume, the access flow and the flow information of a time point are realized, and drawing and charging are realized according to the total access volume, the access flow and the flow information of the time point.
Drawings
FIG. 1 is a schematic diagram of a multi-instance multi-tenant scenario provided by MinIO in the prior art;
FIG. 2 is a block diagram of an embodiment of an object storage architecture for use with the present invention;
FIG. 3 is a schematic diagram of a single instance deployment embodiment of the present invention;
FIG. 4 is a flowchart of a method for implementing public cloud object storage according to the present invention;
FIG. 5 is a schematic diagram of an embodiment of a first rule generated by the present invention;
FIG. 6 is a schematic diagram of an embodiment of creating a data bucket of the present invention;
FIG. 7 is a schematic diagram of an embodiment of the present invention for creating multiple users corresponding to one tenant;
FIG. 8 is a schematic diagram of policy rules of the object storage service according to the present invention;
FIG. 9 is a schematic diagram of an embodiment of a MinIO single-instance multi-tenant scenario provided in the present invention;
FIG. 10 is a schematic diagram of an embodiment of the present invention for creating data compression buckets associated with data buckets;
FIG. 11 is a flow chart illustrating an embodiment of data compression according to the present invention;
FIG. 12 is a flowchart illustrating an embodiment of querying a file list according to the present invention;
FIG. 13 is a flow diagram illustrating an embodiment of statistics on access traffic to a bucket according to the present invention;
FIG. 14 is a schematic diagram of an embodiment of the present invention of a message for creating a bucket http request;
FIG. 15 is a flow chart illustrating the query of the entire traffic of a bucket according to the present invention;
FIG. 16 is a flow chart illustrating the flow of querying a bucket for traffic over a period of time according to the present invention.
Detailed Description
Some abbreviations and key terms mentioned in this disclosure are first described.
: minIO is a high performance object store issued under GNU Affero Universal public license v 3.0. It is an API compatible with Amazon S3 cloud storage services. MinIO is used to build a high performance infrastructure for machine learning, analysis and application data workloads.
Single example
An instance of an object is generated only once throughout the application lifecycle, and all requests (method calls) are handled by this instance.
Multi-tenant
A multi-tenant technology or a multi-lease technology, saaS for short, is a software architecture technology.
The carrier of the object can be understood as a "container" in which the object is stored, and the "container" has no upper limit of capacity, and is referred to as a data bucket in the present application.
High performance HTTP and reverse-proxy web servers.
Erasure code
Erasure Coding (EC) is a method of data protection that segments data into fragments, expands, encodes, and stores redundant data blocks in different locations, such as disks, storage nodes, or other geographic locations.
Object Storage, also referred to as "Object-oriented Storage," English is Object-based Storage. Many cloud vendors now refer directly to as "cloud storage". The Object storage shows a 'bucket', and objects (objects) can be placed in the 'bucket'. This object includes three parts: key, data, metadata. The storage space of the object storage is a Bucket, if the object storage is used for storing data, the Bucket needs to be created first, and attributes such as the region, the access authority and the storage type of the Bucket are specified. The external service is provided in the form of HTTP RESTful API, and different domain names are required for accessing different regions. In an object storage system, you cannot directly open/modify a file, but only download, modify, and then upload the file.
For a further understanding of the invention, reference will now be made to the preferred embodiments of the invention by way of example, and it is to be understood that the description is intended to further illustrate features and advantages of the invention, and not to limit the scope of the claims. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
The invention will now be described in further detail with reference to the accompanying drawing 1 and the specific embodiments thereof
In an embodiment of the present invention, the object storage service is MinIO, the object storage management end is a Web end, and a schematic diagram of deploying a single instance in the public cloud object storage based on MinIO is shown in fig. 3. The flow chart of the implementation method of the public cloud object storage is shown in fig. 4, and specifically comprises the following steps:
s11, responding to the opening of object storage, the object storage management end generates random characters, generates a first rule based on a random character string prefix, and calls an object storage service interface to issue the random characters to the object storage service.
And when the web end receives the tenant opening object storage command, generating a random first character bound with the tenant, for example, generating a random character "ghydtbee-" bound with the tenant. The random characters corresponding to each tenant are different.
S12, the object storage service creates a second rule based on the first rule.
And generating a second rule, namely a policy rule, related to the random character string through the Minio interface according to the first rule issued to the object storage service. Take the example shown in fig. 5.
When the object storage is created, a random first character "ghydtbee-" bound with the tenant is instantiated, and an authority control rule of the corresponding tenant is created, and the authority rule with a wildcard can limit the access resource of the tenant.
S13, responding to the creation of the data bucket, the object storage management end creates a data bucket name based on the random characters and sends the data bucket name to the object storage service.
As shown in fig. 6, the wildcard-based permission rules can restrict the tenant's access to resources. When a tenant creates a bucket, the Web end explicitly allows and only allows the creation of the bucket resource beginning with "gshdhtbee-", so that resource isolation among different tenants is achieved, and naming conflict can never occur when the different tenants create the bucket.
If a bucket with an arbitrary name can be created like other vendors, a conflict may occur to cause a creation failure, and a large number of resource restriction rules need to be configured.
S14, the object storage service judges whether to create the data bucket or not based on the second rule and the data bucket name, if the data bucket name meets the second rule, the data bucket is successfully created, and if not, the data bucket is failed to be created.
In some embodiments, the web side forwards the name of the data bucket bound with the random character to MinIO through the proxy server, and MinIO judges that the name of the data bucket is not legal based on the policy rule, and if the name of the data bucket is legal, the creation of the data bucket is successful. If not, the bucket creation fails.
In some embodiments, a MinIO-based Service account count is also included to simulate different users under the same tenant.
As shown in fig. 7, the Service Account based on MinIO simulates different users under the same tenant, and adds different rules in the Service Account to limit it to access only some buckets. Thus, the user can access the specific buckets with the Service Account and only the Service Account.
S15, the object storage service manages the data bucket based on the second rule.
The policy authority of the tenant bound with the random character 222222-generated by MinIO as shown in fig. 8 allows the corresponding tenant to download objects, list a bucket list, create a bucket and delete a bucket, thereby implementing management of a data bucket.
As shown in fig. 9, the present invention employs a single instance deployment to implement multi-tenancy, consuming only one fifth of the memory occupied in fig. 1, and occupying only one port, which is only the case of 5 tenants. The method aims to solve the problem of resource isolation between tenants brought by single instances. In one embodiment of the invention, when the tenant opens the OSS function, a string of random character string prefixes are generated, the prefixes are bound with the tenant, and Policy rules are generated according to the prefixes, the rules are bound with the tenant, the tenant explicitly uses the prefixes to create and manage bucket resources, and other tenants cannot use the prefixes, so that isolation between the tenants is achieved.
In another embodiment of the present invention, the object storage service of the application further includes a proxy server disposed between the object storage service and the object storage manager, and the proxy server deploys a compression module for compressing the files in the data bucket. The object storage service adopts MinIO, the object storage management end is a web end, and the proxy server adopts Nginx.
Compressing files in a data bucket may be performed by adding a compression module deployed in a proxy server
Now, it can also be implemented directly based on modifying the MinIO code. In an embodiment of the present invention, the compression of data in a data bucket is realized by directly modifying the code in the MinIO. The method is mainly used for compressing the small files which are not changed for a long time and have smaller files and the number of the small files is set, so that the number of file data and metadata is reduced, disk fragments are reduced, the space utilization rate is increased, and a large number of small files in the bucket can be enumerated only by reading header information of a compression packet.
S21, a data compression bucket associated with the data bucket is created based on the second rule, as shown in fig. 10.
While creating a bucket resource prefixed with "ghydbee-", an associated compression bucket is also created, with the bucket name prefixed with "zip _ ghydbee-".
S22, scanning the data bucket based on the preset period, and obtaining file information in the data bucket, where the flow diagram is shown in fig. 11.
The data bucket is scanned periodically to obtain the file size and the last modification time in the data bucket.
And S23, based on the comparison between the acquired file information and a preset compression threshold, packing and compressing the files meeting the preset compression condition, naming the compressed files according to a preset rule, and storing the files into a data compression barrel associated with the data barrel.
The preset threshold values comprise a preset time threshold value compared with unmodified time, a preset size threshold value compared with the size of the file and a preset number threshold value compared with the number of the files. The specific process of compression is as follows.
Obtaining the time of the file which is not changed based on the last modification time and the current time in the obtained file information;
comparing the unchanged time with a preset time threshold, and if the unchanged time is greater than the preset time threshold, taking the file meeting the conditions as a first compressed file set;
if the size of the files in the first compressed file set is smaller than a preset size threshold value, summarizing the files meeting the conditions to a second compressed file set;
and comparing the number of the files in the second compressed file set with a preset number threshold, and if the number of the files in the second compressed file set is greater than or equal to the preset number threshold, taking the files in the second compressed file set as files meeting preset compression conditions.
The packing compression meets the preset compression condition, the compressed file is named according to the preset rule, and the file is generally stored according to the timestamp.
And S24, deleting the compressed files in the data bucket.
Taking a, b, c, d, e and f files existing in a data bucket as an example, the preset size threshold is 1k, the time without change is 10s, and the preset number threshold is 2, if a and b meet the preset compression condition, that is, the preset size threshold is 1k, the time without change is 10s, the preset number threshold is 2, that is, the number of the files smaller than 1k, the time without change is greater than 10s, and more than 2 files meeting the above condition are compressed. A and b are packaged as a zip file prefixed with a time stamp. And store the zip file with the timestamp prefix into the data compression bucket associated with the data bucket. Files a and b are deleted in the data bucket, and c, d, e and f files are reserved.
In another embodiment of the present application, the method further includes querying a file list based on the data compression bucket, and the flowchart is shown in fig. 12, and specifically includes:
s31, responding to the file request for inquiring the data bucket, inquiring the compressed files in the data compression bucket related to the data bucket.
And in the operation of enumerating all files in the bucket, initiating the file list in the data bucket, and then firstly querying the compressed text in the data compression bucket by the object storage service.
And S32, reading a compressed file list in the compressed file based on a preset rule, and reading all file lists in the data bucket.
And reading the file list of all the compressed packages in the data compression bucket according to a preset rule by using Uzip-1. And reading all file lists under the data bucket, and integrating the two file lists.
And S33, returning the summary compressed file list and the file list in the data bucket.
And returning the summarized file list according to a preset format.
It should be noted that, when the compression module deployed in the proxy server is used to compress the file in the data bucket, the compressed data bucket of the file minIO needs to be downloaded to the local where the proxy server is deployed, and after compression, the compressed data bucket is uploaded to the data compression bucket of the minIO.
In another embodiment of the present application, the method further includes deploying a statistics module for counting access traffic of the data bucket based on the proxy server, and a flowchart is shown in fig. 13, which specifically includes:
s41, the statistic module creates a bucket data table for storing data bucket information.
A barrel data table is created in the memory of the proxy server, and the data table is empty when the data table is started. The information corresponding to each data bucket stored in the bucket data table, the total length of the message corresponding to each bucket, the total access amount and the last refreshing time.
And S42, the proxy server receives the message and forwards the message to the statistical module.
The flow stored by the access object firstly enters the proxy server and then is forwarded through the proxy server, for example, after the proxy server receives the message, the message is forwarded to the statistical module for statistics, and then is forwarded out.
S43, the statistical module obtains the message information, and searches whether the data bucket name in the message exists or not based on the bucket data table, if not, the corresponding data bucket information in the message is created in the bucket data table.
Taking the message information of the http request for creating the bucket shown in fig. 14 as an example, the bucket name and the message length information in the request content may be obtained. And searching whether the data bucket information exists in the table according to the bucket name, wherein the data bucket information does not exist in the bucket data table.
And S44, storing the message length and the access amount of the corresponding data bucket in the message to a bucket data table.
Adding the length of the acquired message in the data bucket information, and adding 1 to the access amount.
And S45, storing the information in the data bucket table to a file corresponding to the data bucket based on the current time and the preset period.
And judging the last refreshing time of the barrel and the current time interval, if the preset period is reached, storing the information in the barrel data table into the local text of the corresponding barrel, and simultaneously storing the timestamp.
In some embodiments, the text corresponding to the data buckets is deployed on a server deploying the proxy server.
In some embodiments, after a reboot, the text statistics of the bucket are collected and loaded into a bucket data table in memory. The text name of the text corresponding to the data bucket is the bucket name, bucket information is created in the memory table according to the bucket name, then the last line of the text corresponding to the data bucket is loaded to initialize the total length of the message and the total access amount, and the refreshing time is the current time.
And S46, responding to the query condition, and formatting the data corresponding to the query condition in the return file.
The statistical module provides an interface for external inquiry, the inquiry unit is a bucket, and when a certain bucket is called and inquired, the last line of the text of the corresponding bucket is read, so that the total access volume and the access flow are obtained for flow charging. The schematic view is shown in fig. 15.
In some embodiments, if the access flow and the access amount of the corresponding time point are to be drawn, each line of the text is read, the current line is subtracted by the previous line to obtain the access data per second, the total information and the information per second are subjected to certain format response to the requester, and then the requester can draw the information drawing according to the response.
In some embodiments, if the query condition is to query only data for a period of time, then the data for the specified period of time is loaded from the text according to the query condition, and the flow is shown in fig. 16.
According to the method for realizing the public cloud object storage, the MinIO is used for realizing the isolation of different tenant resources in the public cloud, and the excessive resource consumption is reduced. And the small files are compressed through compression of a certain rule, so that the file list is enumerated quickly, and the query performance and the disk utilization rate are increased. And the charging can be accurately carried out based on the flow of each bucket and the requested quantity, and the charging problem of the storage of the public cloud object is realized.
In addition, the basic principles and preferred embodiments of the present invention have been described above, and many changes and modifications may be made by those skilled in the art based on the above description, which should fall within the scope of the present invention.
Claims (9)
1. A method for realizing public cloud object storage is applied to an object storage architecture at least deploying object storage service and an object storage management end, wherein the object storage service deploys a single process instance, and the method for realizing the public cloud object storage specifically comprises the following steps:
responding to the opening of object storage, generating a random character by an object storage management end, generating a first rule based on a prefix of a random character string, and calling an object storage service interface to issue to an object storage service;
the object storage service creating a second rule based on the first rule;
in response to the creation of the data bucket, the object storage management terminal creates a data bucket name based on the random characters and issues the data bucket name to the object storage service;
and the object storage service judges whether to create the data bucket based on the second rule and the data bucket name, if the data bucket name meets the second rule, the creation of the data bucket is successful, and if not, the creation of the data bucket fails.
2. The method of claim 1, further comprising the object storage service managing the data buckets based on a second rule.
3. The method for implementing public cloud object storage according to claim 1, wherein the object storage architecture of the application further includes a proxy server disposed between the object storage service and the object storage management end, and the proxy server deploys a compression module for compressing files in the data buckets and/or deploys a statistics module for counting access traffic of the data buckets.
4. The method for implementing public cloud object storage according to claim 3, wherein the object storage service further creates a data compression bucket associated with a data bucket, or creates a data compression bucket associated with a data bucket based on a create data bucket request forwarded by the compression module.
5. The method for implementing public cloud object storage according to claim 4, further comprising compressing the small file based on a data compression bucket, specifically comprising:
scanning a data bucket based on a preset period, and acquiring file information in the data bucket;
based on the comparison between the acquired file information and a preset compression threshold, packing and compressing the files meeting the preset compression condition, naming the compressed files according to a preset rule, and storing the files into a data compression barrel associated with the data barrel;
and deleting the compressed files in the data bucket.
6. The method for implementing public cloud object storage according to claim 5, wherein the compressing a file meeting a preset compression condition based on comparison between the acquired file information and a preset compression threshold specifically comprises:
obtaining the time of the file which is not changed based on the last modification time and the current time in the obtained file information;
comparing the unchanged time with a preset time threshold, and if the unchanged time is greater than the preset time threshold, taking the file meeting the conditions as a first compressed file set;
if the size of the files in the first compressed file set is smaller than a preset size threshold value, summarizing the files meeting the conditions to a second compressed file set;
and comparing the number of the files in the second compressed file set with a preset number threshold, and if the number of the files in the second compressed file set is greater than or equal to the preset number threshold, taking the files in the second compressed file set as files meeting preset compression conditions.
7. The method for implementing public cloud object storage according to claim 5, further comprising querying a file list based on the data compression bucket, specifically comprising:
responding to a file request for inquiring the data bucket, and inquiring the compressed files in the data compression bucket;
reading a compressed file list of the compressed files based on a preset rule, and reading all file lists in the data bucket;
a summary compressed file list and a list of files in the data bucket are returned.
8. The method for implementing public cloud object storage according to claim 3, further comprising deploying a statistics module based on the proxy server to count access traffic of the data bucket, specifically comprising:
a counting module creates a barrel data table used for storing data barrel information;
the proxy server receives the message and forwards the message to the statistical module;
the counting module acquires message information, searches whether a data bucket name in the message exists or not based on a bucket data table, and if not, creates corresponding data bucket information in the message in the bucket data table;
storing the message length and the access amount of a corresponding data bucket in the message to a bucket data table;
storing the information in the bucket data table to a file corresponding to the data bucket based on the current time and a preset period;
and responding to the query condition, and formatting data corresponding to the query condition in the return file.
9. The method for implementing public cloud object storage according to claim 8, wherein the formatting data corresponding to the query condition in the return file in response to the query condition specifically includes:
based on the query conditions, the statistical module queries the files corresponding to the data bucket, reads the information corresponding to the query conditions in the files,
and obtaining the timestamp, the flow of the preset period, the access amount and the bucket flow of the time period based on the information in the file, and returning according to the preset format.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211587570.6A CN115587390B (en) | 2022-12-12 | 2022-12-12 | Method for realizing public cloud object storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211587570.6A CN115587390B (en) | 2022-12-12 | 2022-12-12 | Method for realizing public cloud object storage |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115587390A true CN115587390A (en) | 2023-01-10 |
| CN115587390B CN115587390B (en) | 2023-03-10 |
Family
ID=84783365
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211587570.6A Active CN115587390B (en) | 2022-12-12 | 2022-12-12 | Method for realizing public cloud object storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115587390B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103218175A (en) * | 2013-04-01 | 2013-07-24 | 无锡成电科大科技发展有限公司 | Multi-tenant cloud storage platform access control system |
| US20140289391A1 (en) * | 2013-03-19 | 2014-09-25 | Cognizant Technology Solutions India Pvt. Ltd. | Framework for facilitating implementation of multi-tenant saas architecture |
| CN113886331A (en) * | 2021-12-03 | 2022-01-04 | 苏州浪潮智能科技有限公司 | Distributed object storage method and device, electronic equipment and readable storage medium |
| CN114327249A (en) * | 2020-10-10 | 2022-04-12 | 华为云计算技术有限公司 | Data modification method of object storage service, service node, client and equipment |
| WO2022127762A1 (en) * | 2020-12-14 | 2022-06-23 | 华为云计算技术有限公司 | Cloud platform and bucket management method for object storage service provided thereby |
| CN115150419A (en) * | 2022-09-05 | 2022-10-04 | 杭州华卓信息科技有限公司 | Configuration and access method and system for hybrid cloud object storage |
-
2022
- 2022-12-12 CN CN202211587570.6A patent/CN115587390B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140289391A1 (en) * | 2013-03-19 | 2014-09-25 | Cognizant Technology Solutions India Pvt. Ltd. | Framework for facilitating implementation of multi-tenant saas architecture |
| CN103218175A (en) * | 2013-04-01 | 2013-07-24 | 无锡成电科大科技发展有限公司 | Multi-tenant cloud storage platform access control system |
| CN114327249A (en) * | 2020-10-10 | 2022-04-12 | 华为云计算技术有限公司 | Data modification method of object storage service, service node, client and equipment |
| WO2022127762A1 (en) * | 2020-12-14 | 2022-06-23 | 华为云计算技术有限公司 | Cloud platform and bucket management method for object storage service provided thereby |
| CN113886331A (en) * | 2021-12-03 | 2022-01-04 | 苏州浪潮智能科技有限公司 | Distributed object storage method and device, electronic equipment and readable storage medium |
| CN115150419A (en) * | 2022-09-05 | 2022-10-04 | 杭州华卓信息科技有限公司 | Configuration and access method and system for hybrid cloud object storage |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115587390B (en) | 2023-03-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107861686B (en) | File storage method, server and computer readable storage medium | |
| CN108984560B (en) | File storage method and device | |
| CN110324177B (en) | Service request processing method, system and medium under micro-service architecture | |
| US6804719B1 (en) | Method and system for relocating files that are partially stored in remote storage | |
| KR101948956B1 (en) | Method for data maintenance | |
| CN104951474A (en) | Method and device for acquiring MySQL binlog incremental logs | |
| CN111400312B (en) | Edge storage database based on improved LSM tree | |
| CN108776682B (en) | Method and system for randomly reading and writing object based on object storage | |
| CN107483631B (en) | Method for controlling cache to realize mobile internet service access | |
| CN102411598A (en) | Method and system for realizing data consistency | |
| US20200142634A1 (en) | Hybrid distributed storage system to dynamically modify storage overhead and improve access performance | |
| CN103716346A (en) | Management method and device of application on android handset client | |
| CN113923213B (en) | Directory-level service quality control method, system and access node | |
| WO2014161261A1 (en) | Data storage method and apparatus | |
| WO2017174013A1 (en) | Data storage management method and apparatus, and data storage system | |
| CN109885577A (en) | Data processing method, device, terminal and storage medium | |
| CN113806300A (en) | Data storage method, system, device, equipment and storage medium | |
| CN115587390B (en) | Method for realizing public cloud object storage | |
| WO2023160276A1 (en) | Data processing method and apparatus, computer device and readable storage medium | |
| CN107479827A (en) | A kind of mixing storage system implementation method based on IO and separated from meta-data | |
| CN114942908B (en) | Index system, data processing method, electronic device, and medium | |
| CN113296714B (en) | Data storage system based on NAS protocol | |
| CN115955581A (en) | Real-time video processing method, device, equipment and storage medium | |
| US20050188380A1 (en) | Cache control device, and method and computer program for the same | |
| CN113032414B (en) | Data management method, device, system, computing equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: 310000 room 611-612, Zhuoxin building, No. 3820, South Ring Road, Puyan street, Binjiang District, Hangzhou City, Zhejiang Province Patentee after: Hangzhou Youyun Technology Co.,Ltd. Country or region after: China Address before: 310000 room 611-612, Zhuoxin building, No. 3820, South Ring Road, Puyan street, Binjiang District, Hangzhou City, Zhejiang Province Patentee before: Hangzhou Youyun Technology Co.,Ltd. Country or region before: China |