CN106775499A - It is a kind of that public memory space is carried out to rationalize exposed method, exposure control system and a kind of cloud storage system - Google Patents
It is a kind of that public memory space is carried out to rationalize exposed method, exposure control system and a kind of cloud storage system Download PDFInfo
- Publication number
- CN106775499A CN106775499A CN201710067189.XA CN201710067189A CN106775499A CN 106775499 A CN106775499 A CN 106775499A CN 201710067189 A CN201710067189 A CN 201710067189A CN 106775499 A CN106775499 A CN 106775499A
- Authority
- CN
- China
- Prior art keywords
- module
- storage space
- access
- access request
- public
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000012545 processing Methods 0.000 claims description 19
- 238000013500 data storage Methods 0.000 claims description 15
- 239000012634 fragment Substances 0.000 claims description 9
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000013475 authorization Methods 0.000 claims description 4
- 230000011218 segmentation Effects 0.000 claims description 3
- 230000006872 improvement Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000013467 fragmentation Methods 0.000 description 4
- 238000006062 fragmentation reaction Methods 0.000 description 4
- 230000005012 migration Effects 0.000 description 4
- 238000013508 migration Methods 0.000 description 4
- 238000013507 mapping Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 2
- 239000003999 initiator Substances 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000003466 anti-cipated effect Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0626—Reducing size or complexity of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0608—Saving storage space on storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Public memory space is carried out to rationalize exposed method, exposure control system and its cloud storage system the invention provides a kind of, the method includes：The pre- structure access consideration in card module；Access request is received and parsed through by constraints module, and calls the access consideration in card module, and row constraint only is entered to the access request for meeting access consideration；By statistical module counts and resource occupation degree of the access request to public memory space is updated, and submit the Realtime Statistics for characterizing public memory space to the provider of public memory space；The provider of public memory space regularly updates according to Realtime Statistics to the access consideration in card module.By the present invention, realize and public memory space is reasonably exposed to user, the utilization rate of the residual memory space that bucket is formed is improve, the cost of serving and provider that effectively reduce the deployment side for providing publicly-owned cloud service dispose the lower deployment cost of public memory space.
Description
Technical Field
The invention relates to the technical field of storage systems, in particular to a method for reasonably exposing a public storage space, an exposure control system for reasonably exposing the public storage space based on the method, and a cloud storage system comprising the exposure control system for reasonably exposing the public storage space.
Background
The public cloud refers to cloud services which can be used and are provided for users by service providers of the public cloud, the public cloud can be generally used through the internet, and the core attribute of the public cloud is shared resource services. The cloud storage system is used as an important component of a public cloud and is mainly used for undertaking tasks such as data storage, data access and application access. A service provider providing a public cloud deploys the public cloud service in a physical server of an operator providing a cloud platform service and opens the right to access data and/or applications to the public (i.e., a user or user of the public cloud).
The cloud storage system can provide unlimited storage capacity on a user level. However, such unlimited permission opening may result in that the storage resources of the cloud storage system leased by the service provider are excessively occupied by the user. Since the cloud storage space formed by the cloud storage system is purchased by the service provider for a fee, the situation that the storage resources are excessively occupied also results in that the service provider cannot provide healthy storage services or applications to all users fairly, thereby causing a reduction in user experience when the users use the public cloud. With the rise of wechat public platforms, internet finance and other public services, the above-mentioned risks faced by service providers (i.e. deployers) offering the above-mentioned public services or customized services are also increasing.
The chinese patent application publication No. CN101799797A discloses a "dynamic allocation method of user disk quotations in a distributed storage system," which provides a disk quota management scheme for a distributed storage service system based on user access behavior correlation. Although the prior art can control the size of the storage space used by each user, the utilization rate of the data storage server can be improved by dynamically adjusting the quota. But is essentially to solve the problem of fair sharing of data storage servers between users. But this solution is not applicable to public cloud storage systems. Because, in a public cloud storage system, any user is allowed to store any large amount of data. Therefore, the method does not solve the problem of data sharing, and the reasonable management and control of the public storage space exposed by the public storage space by the service provider in the public cloud platform cannot be realized. Therefore, the above prior art cannot realize the rationalization and control of the public storage space exposed to the user by the provider of the service in the public cloud platform.
In view of the above, there is a need to improve the technology of reasonably exposing the common storage space in the prior art to solve the above problems.
Disclosure of Invention
The invention aims to disclose a method for reasonably exposing a public storage space, which is used for reasonably controlling the public storage space exposed to a user by a deployment party, reducing the service cost of the deployment party, avoiding huge fund waste caused by excessively exposing the public storage space, improving the tolerance of the public storage space provided by a provider to abnormal large files, improving the user experience of the user using public cloud service, improving the utilization rate of a storage barrel and avoiding storage space fragments by reasonably exposing the public storage space; meanwhile, the method also discloses an exposure control system applying the control method and a cloud storage system comprising the exposure control system.
To achieve the first object, the present invention provides a method for rationalizing exposure of a common storage space, which is characterized by comprising:
pre-constructing access conditions in the plug-in module;
the constraint module receives and analyzes the access request, calls the access condition in the plug-in module and only constrains the access request meeting the access condition;
the statistical module is used for counting and updating the resource occupation rate of the access request to the public storage space and submitting real-time statistical data representing the public storage space to a provider of the public storage space;
the provider of the public storage space periodically updates the access conditions in the plug-in modules based on the real-time statistics.
As a further improvement of the invention, the plug-in module analyzes the Authorization field and the Host field of the HTTP request to count the access indexes generated by the access request sent by the user accessing the public storage space.
As a further improvement of the invention, the access indicators include requester attributes, a path of the access request, an address of a common memory space to which the access request is directed, and a data stream.
As a further improvement of the present invention, the method further comprises: and formatting the real-time statistical data, generating formatted data in an HTML format, and reporting the formatted data to a provider.
As a further improvement of the present invention, the method further comprises the steps of judging whether an access request for accessing the public storage space is submitted by a deployment party through a constraint module;
when the deployment party submits the access request, the access request is not restricted;
when the user submits the access request, the access condition in the plug-in module is called, and only the access request meeting the access condition is restricted.
As a further improvement of the invention, the access condition comprises the access times, the data flow, the access time and the identity information of the user to the independent data storage space in the public storage space.
As a further improvement of the present invention, the updating is specifically to execute a nginx-s reload instruction to load the latest LUA file.
As a further improvement of the present invention, the method further comprises:
counting the residual storage space in the public storage space and reporting to a provider;
the provider persists the residual storage space of the plurality of buckets in the common storage space to take the screened buckets which are not fully stored as candidate buckets;
and the file boot storage corresponding to the access request submitted by the user at the next time is put into the matched candidate storage bucket.
As a further improvement of the present invention, the method further comprises: dividing the file corresponding to the access request submitted next time by the user according to a set division strategy;
the segmentation strategy is as follows: the size of one of the file fragments formed through the division is matched with the residual storage space of the candidate buckets in the common storage space.
To achieve the second object, the present invention further discloses an exposure control system for rationalizing exposure to a common storage space, the exposure control system comprising:
a statistic module composed of a statistic request monitoring module, a data processing module and a data receiving module,
a plug-in module composed of a request constraint module and an index statistic module,
the constraint module consists of a constraint request monitoring module, a constraint judging module and a constraint executing module; wherein,
the constraint module is used for receiving and analyzing the access request, calling the access condition in the plug-in module and only constraining the access request meeting the access condition,
the statistical module is used for counting and updating the resource occupation rate of the access request to the public storage space and submitting real-time statistical data representing the public storage space to a provider of the public storage space,
the plug-in module is used for storing the access conditions and receiving updates of the access conditions in the plug-in module by a provider of the public storage space periodically according to the real-time statistical data.
In addition, to achieve the last object of the present invention, the present invention also discloses a cloud storage system, comprising:
a number of storage components located in different geographical areas,
a logical processing cluster connected to the storage component,
the reverse-proxy cluster, and,
an exposure control system connecting the logic processing cluster and the reverse proxy cluster;
the exposure control system is configured as a system according to the method of any of the inventions described above.
Compared with the prior art, the invention has the beneficial effects that: by the method and the device, the public storage space is reasonably exposed to the user, the utilization rate of the residual storage space formed by the storage barrel is improved, and the service cost of a deployment party providing public cloud service and the deployment cost of a provider deploying the public storage space are effectively reduced.
Drawings
Fig. 1 is an architecture diagram of a cloud storage system based on a distributed storage architecture in the prior art;
FIG. 2 is an architecture diagram of a cloud storage system in an embodiment of the present invention;
FIG. 3 is an architectural diagram of the exposure control system of FIG. 2;
FIG. 4 is a flowchart illustrating dynamic adjustment of the public storage space exposed to the user by the exposure control system;
FIG. 5 is a flowchart illustrating dynamic adjustment of a public storage space exposed to a user by an exposure control system when an access request is required to segment a corresponding large file;
fig. 6 is a distribution diagram of data of customers of the deployer a and the deployer B stored in a plurality of buckets of a common storage space in a certain state;
FIG. 7 is an exemplary diagram of writing a file A-4 corresponding to an access request initiated by a client of deployer A into a matching candidate bucket in the common storage space based on the example of FIG. 6;
FIG. 8 is an exemplary diagram of a file A-5 corresponding to an access request initiated by a client of the deployer A and requiring to partition the file A-5 and write the partitioned file A-5 into two candidate buckets in a common storage space.
Detailed Description
The present invention is described in detail with reference to the embodiments shown in the drawings, but it should be understood that these embodiments are not intended to limit the present invention, and those skilled in the art should understand that functional, methodological, or structural equivalents or substitutions made by these embodiments are within the scope of the present invention.
Before describing the present invention in detail, first, terms in the embodiments of the specification are defined.
Term "Storage bucket": the virtual storage unit is composed of a physical server or a computer, has data storage capacity and is relatively independent. Typically, a plurality of buckets are included in a public cloud environment, each bucket having independent data storage capacity and having an upper limit value of storage space associated with a disk specification in a physical server. In this specification, a bucket is synonymous with "QUOTA".
Term "Public storage space": multiple buckets in a public cloud environment provide access to the public cloud or to all users open access to various platforms/applications deployed in the public cloudA virtualized storage space of rights. A user can perform various data operations such as read operation, write operation, delete operation or migration operation on one bucket or a plurality of buckets in the public storage space through the Restful API.
Term "Provider side", refers to the party that provides the hardware device and forms the common storage space through virtualization technology.
Term "Deployment party"refers to a party of various applications (e.g., cloud disk, wechat public platform) with data access capability built or deployed or formed in a public storage space provided by a provider, which pays a fee to purchase several buckets, and provides various data operations such as read operation, write operation, delete operation or migration operation to a general internet user or APP user through the storage capability of the buckets for data and in conjunction with a background program.
Term "User side"refers to a party performing various data operations such as read operation, write operation, delete operation, or migration operation in a data bucket by sending an access request, and is generally a common internet user or APP user (e.g., a cloud disk user, a WeChat user, an email user, etc.). It is reasonably anticipated by one of ordinary skill in the art that the APPs in the embodiments of the present description include, but are not limited to: wechat, microblog, blog, QQ, strange, Facebook, kakko, instrgram, twitter, or other various public service platforms with data interaction functionality, and other types of instant chat software.
Before describing the embodiments in detail, a brief explanation of a cloud storage system, which is the mainstream in the prior art, is provided. Currently, distributed Storage systems based on an Object Storage (Object Storage) architecture, which is the mainstream technology, are the mainstream network Storage technology.
Referring to fig. 1, in the object storage, a cloud storage system generally includes a reverse proxy cluster 10, which is constructed with a plurality of gateways, of which only a gateway 101 and a gateway 102 are shown in this specification for exemplary illustration; a logic processing cluster 20 for executing logic processing services, which is constructed with a plurality of logic servers, only a logic server 201 and a logic server 202 are shown in the present specification; metadata clusters 30 and data storage clusters 40. Metadata cluster 30 illustrates metadata server 301 and metadata server 302, and data storage cluster 40 illustrates data storage server 401 and data storage server 402.
The gateway in the reverse proxy cluster is responsible for receiving an access request to the public storage space sent by a user (i.e. a user), and performs a load balancing operation to send the access request to the back-end logical processing cluster 20 according to a set distribution rule. The logic processing cluster 20 stores the metadata and the file data of the file corresponding to the access request in the metadata cluster 30 and the data storage cluster 40 at the back end, respectively. The deployer purchases a number of buckets from the provider in response to the file or data corresponding to the access request sent by the consumer. In a public cloud environment, traffic charges generated in the process that a user accesses a storage bucket of a component of a deployer are borne by the deployer, and technical support of back-end storage is completed by a provider.
Data operations of reading data, writing data, deleting data and the like are formed in one bucket. The deployer may restrict the various operations of the user on the data in a certain bucket by setting permissions. However, such a constraint mechanism is rigid and cannot be applied to providing personalized storage service for public clouds by users. In order to charge users using public storage space of public cloud, a traditional cloud storage system cannot count the activity degree of each storage bucket, so that a deployer cannot accurately adjust access policies, migration policies and the like of data in the storage buckets, a provider or the deployer cannot delete invalid data and dead data, and the storage efficiency of the storage buckets on the data and the user experience of the users are seriously affected. However, if the access of the user is excessively restricted, the access of other users to the bucket may be caused, and the user experience of the client (i.e., the user) of the subscriber may be seriously degraded.
The first embodiment is as follows:
referring to fig. 2 and fig. 3, the embodiment discloses an exposure control system 50 for performing rational exposure on a common storage space, which is connected between a reverse proxy cluster 10 and a logical processing cluster 20, so as to perform rational exposure on a storage space of a bucket of the common storage space exposed to a user through the exposure control system 50 and perform rational guidance on data written into the bucket by the user, so as to avoid written data which is incomplete in each bucket and avoid fragmentation of the storage space.
Specifically, the exposure control system 50 includes: a statistic module 51 composed of a statistic request monitoring module 511, a data processing module 512 and a data receiving module 513; a plug-in module 52 consisting of a request constraint module 521 and an index statistics module 522; and a constraint module 53 comprising a constraint request monitoring module 531, a constraint judging module 532 and a constraint executing module 533. The constraint module 53 is configured to receive and parse the access request, invoke the access condition in the plug-in module 52, and constrain only the access request meeting the access condition. The statistic module 51 is configured to count and update the resource occupancy of the access request to the public storage space, and submit real-time statistic data representing the public storage space to a provider of the public storage space. The plug-in module 52 is used to store the access conditions and receive updates from the provider of the common storage space to the access conditions in the plug-in module 52 based on real-time statistics on a regular basis.
The statistics request snooping module 511 receives a statistics request sent by the provider for counting the number of buckets in the common storage space and the remaining storage space of each bucket, and receives a return structure of the data processing module 512, so as to provide a decision for the provider to update the access condition of the plug-in module 52. The data processing module 512 may receive the statistics from the data receiving module 513 and format the statistics to generate HTML formatted data for listening through the statistics request module 511 and reporting to the provider. The data receiving module 513 is responsible for receiving the statistics of each access request constraint sent by the metric statistics module 522 in the plug-in module 52.
In this embodiment, different plug-in modules may be deployed to different users. Since the access request sent by the user passes through the index statistics module 522 each time, the index statistics module 522 can determine, by analyzing the Authorization field and the Host field of the HTTP request, who the user corresponding to the access request specifically belongs to, and to which issuer the access request originated by the user is assigned to the bucket to which the provider is assigned. The index counting module 522 counts parameters such as the number of accesses to a bucket, the name of a user, and data streams generated by accessing a specific bucket, so as to accurately measure the cost of the user for using the public cloud service. The request restriction module 521 compares the statistical results obtained by the index statistics module 522 on the remaining storage spaces of all the buckets in the common storage space with the pre-configured access conditions, so as to screen out the access request meeting the access conditions or determine whether the access request of the party needs to be restricted. At the same time, the pre-configured access conditions may be dynamically updated.
The constraint request snooping module 531 receives an access request to a bucket from a user or a subscriber forwarded from the gateway 101 in the reverse proxy cluster 10. In this case, the restriction module 53 can determine whether the access request is from the user or the subscriber, and restrict the access request only from the user. The constraint judging module 532 parses the access request and calls the real-time plugin module 52 matching the access request to determine whether the access request needs to be terminated in advance according to the real-time plugin module 52 updated last time. The constraint execution module 533 receives the constraint instruction of the constraint judgment module 532, and is responsible for specifically executing the constraint operation.
The exposure control system 50 composed of the statistical module 51, the plug-in module 52 and the constraint module 53 shown in this embodiment can satisfy personalized and reasonable exposure of the public storage space exposed to the user by the deployment party without changing the architecture of the cloud storage system based on the subtended storage, effectively prevent the problem that some or some storage buckets in the public storage space are excessively occupied by the malicious user on the premise of not increasing the cost of the deployment party, and take into account the use experience of other users in the public cloud platform.
In this embodiment, the statistical module 51, the plug-in module 52, and the constraint module 53 may all be developed based on nginnx (an open source software), so as to implement personalized and customized development on the storage index and the constraint index of the storage bucket in the public storage space, and perform personalized constraint on the pre-configured access condition in the plug-in module 52 according to the reference indexes such as the usage habit of the user and the business credit. Meanwhile, due to the adoption of open source software development, hot plug can be carried out among all the modules, mutual interference among all the modules is avoided, and the stability and reliability of the exposure control system 50 are improved.
Example two:
referring to fig. 3 and 4 in combination, the present embodiment discloses a method for rationalizing exposure of a common storage space, the method including: access conditions are pre-configured in the plug-in module 52; the constraint module 53 receives and analyzes the access request, calls the access condition in the plug-in module 52, and only constrains the access request meeting the access condition; the statistical module 51 counts and updates the resource occupation rate of the access request to the public storage space, and submits real-time statistical data representing the public storage space to a provider of the public storage space; the provider of the common storage space periodically updates the access conditions in the plug-in modules 52 based on real-time statistics.
Specifically, the plug-in module 52 analyzes the Authorization field and the Host field of the HTTP request to count the access index generated by the access request sent by the user accessing the public storage space. The access metrics include requestor attributes, a path of the access request, an address of a common memory space to which the access request is directed, and a data stream. Then, the data processing module 512 may receive the real-time statistical data from the data receiving module 513 and format the statistical data to generate data in HTML format for listening to the module 511 through the statistical request and reporting to the provider.
The method also comprises the steps of judging whether an access request for accessing the public storage space is submitted by a deployment party or not through a constraint module; when the deployment party submits the access request, the access request is not restricted; when the user submits the access request, the access condition in the plug-in module is called, and only the access request meeting the access condition is restricted. The access condition comprises the access times, the data flow, the access time and the identity information of the user to the independent data storage space in the public storage space. The updating is specifically to execute the nginx-s reload instruction, and load the latest LUA file, so as to implement the updating operation of the access condition pre-configured in the plug-in module 52 or the access condition corresponding to the last access request by the provider.
Referring to fig. 4, a method for rationalizing and exposing a common storage space according to the second embodiment is more specifically as follows.
Step S100: an access request is sent. In the engineering example, the access request forwarded by the gateway 101 is sent to the constraint request listening module 531 in the constraint module 53.
Step S101: an access request is received and parsed.
Step S102: the constraint judging module 532 calls the access condition pre-configured in the plug-in module 52.
Step S103: the constraint judging module 532 judges who the initiator of the newly received access request is, if the access request is initiated by a deployment party (such as a cloud disk service provider), or initiated by a use party (such as a general consumer of a cloud disk); if the access request is initiated by the subscriber, the direct jump execution step S104: switching to a logic processing service; if the access request is initiated by the user, step S105 is executed in sequence.
Step S105: judging whether the access request meets a preconfigured access condition, and if not, skipping to execute the step S104; if yes, step S106 is executed in sequence.
Step S106: and counting and updating the occupancy rate of the access request to the public storage space. The occupancy rate reflects the storage state of the plurality of buckets in the common storage space at a specific moment, so that accurate basis and decision can be provided for reasonably guiding the corresponding file or data into the specific bucket for the next initiated access request by updating the occupancy rate.
Step S107: and restricting the data corresponding to the access request according to the pre-constructed access condition.
Step S108: and submitting real-time statistical data representing the public storage space to a provider. The real-time statistical data comprises: the access times, data flow, access time and user identity information of the independent data storage space in the public storage space. Based on the architecture of the subtended storage, a file can be written into a specified bucket under the direction of metadata (descriptive data) held in a metadata server in the metadata cluster 30.
Step S111: the file is written to the bucket. That is, after a larger file is split, two or more fragments of the file may be written into buckets (i.e., candidate buckets) that still have storage capacity, so as to avoid fragmentation of the storage space of the buckets.
Step S112: the access conditions are updated periodically and fed back to the provider. Specifically, the access conditions may be updated at 60 minute intervals.
Step S113: the constraint module determines whether to terminate the access request according to a constraint result; if yes, step S114 is executed in sequence: EDN; if not, the step S104 is executed.
Example three:
referring to fig. 5, fig. 6 and fig. 7, the present embodiment discloses a method for rationalizing exposure of a common storage space, and the main difference between the present embodiment and the second embodiment is that, in the present embodiment, the method further includes: counting the residual storage space in the public storage space and reporting to a provider; the provider persists the remaining storage space of the plurality of buckets in the common storage space to use the screened out unfilled buckets as candidate buckets. And the file boot storage corresponding to the access request submitted by the user at the next time is put into the matched candidate storage bucket. Specifically, refer to step S109 and step S110 in fig. 5.
Referring to fig. 6 and 7, a plurality of buckets are formed by physical server virtualization at the back end of the subtended storage based cloud storage system, and the buckets jointly form a common storage space for providing public cloud services. The storage in the common storage space after the last access request is completed is shown in fig. 6. Wherein, the storage bucket 61, the storage bucket 62 and the storage bucket 63 respectively store the files A-1, A-2 and A-3 written by the client (i.e. the user) of the company of the deployment party A; and documents B-1, B-2, and B-3 written by a client (i.e., a user) of the company of the distributor B. Buckets 64 through 6i through 6j (the number is not limited and depends on the size of the physical server) have no data written to them or any files saved.
Referring to fig. 7, when a file a-4 corresponding to an access request sent by a user needs to be written into a common storage space, there are multiple write paths. In this embodiment, by introducing a mechanism for counting and evaluating the remaining storage space of each bucket, it is possible to write the file a-4 into the bucket that most matches the file size of the file a-4. After responding to the previous access request, none of the buckets 61, 62, and 63 are completely full, and a certain amount of remaining storage space is formed.
In this embodiment, the index statistics module 522 may obtain the remaining storage space of each bucket according to the data read-write status of all buckets in the common storage space, and use a bucket with a certain data storage capacity as a candidate bucket for responding to the data write operation corresponding to the next access request. In FIG. 7, the remaining storage space of bucket 63 matches the size of file A-4. Thus, file a-4 may be written to bucket 63 (i.e., path 1), thereby avoiding blind writing of file a-4 to bucket 64, and avoiding fragmentation of storage space among multiple buckets in a common storage space, to improve subsequent write-ability for large files.
The data write operation corresponding to the next access request follows the following mechanism.
If the residual storage space of all candidate buckets is larger than the size of the file needing to be written, selecting the candidate bucket with the minimum residual storage space to execute the operation of writing the file;
if the remaining storage space of a candidate bucket is partially larger than the size of the file to be written, but partially smaller than the size of the file to be written, then the candidate bucket that best matches the size of the file to be written will be selected.
The name of the candidate bucket is mapped to the bucket selected by the last access request by modifying the POST header of the HTTP request, and the mapping relation is stored. Wherein, the mapping relation is as follows:bucket name/file name → newly selected memory Name of storage barrel. When the initiator (i.e., the user) of the access request sends the access request, the correct file can be read from the bucket through the mapping relationship, and then the HTTP request is forwarded to the logical processing cluster 20.
Example four:
referring to fig. 5 and 8, the present embodiment discloses a method for rationalizing and exposing a common storage space, and the main difference between the present embodiment and the second and/or third embodiments is that, in the present embodiment, the method further includes: and carrying out file division on the file A-5 according to the set division strategy on the file corresponding to the access request submitted next time by the user. The segmentation strategy is as follows: the size of one of the file fragments formed through the division is matched with the residual storage space of the candidate buckets in the common storage space.
In the present embodiment, splitting a file is performed based on the following policy.
First, if the sum of the remaining storage spaces formed by the candidate buckets respectively is equal to the size of the large file a-5, the large file a-5 is subjected to a file splitting operation according to the size of the remaining storage spaces formed in the candidate buckets respectively, and the remaining storage spaces in the candidate buckets 61 to 63 are all filled.
Secondly, if the sum of the residual storage spaces formed by the candidate buckets is larger than the size of the large file A-5, the large file A-5 is subjected to at least one file division in sequence according to the specification of the residual storage spaces formed in the candidate buckets from small to large according to the principle that the candidate buckets with smaller residual storage spaces are written with priority.
Referring to FIG. 8, in the present embodiment, a large file A-5 is partitioned into a file fragment A-5-1 and a file fragment A-5-2, and then the file fragment A-5-2 is written into a candidate bucket 63 (i.e., path 4), and then the fragmented file A-5-1 is written into a candidate bucket 61 (i.e., path 3). The reason candidate bucket 61 is selected without selecting candidate bucket 62 is that the remaining storage space of candidate bucket 61 is less than the remaining storage space of candidate bucket 62, and the remaining storage space of candidate bucket 61 is greater than or equal to the size of file fragment a-5-1. If fragmented file A-5-1 is written into candidate bucket 62, then only a large fragmentation of storage space will result.
Example five:
the embodiment discloses a cloud storage system, which includes: a number of storage components located in different geographical areas, a logical processing cluster 20 connected to the storage components, a reverse proxy cluster 10, and an exposure control system 50 connecting the logical processing cluster 20 with the reverse proxy cluster 10. The exposure control system 50 in this embodiment is configured to execute the system of the method for performing the rationalized exposure of the common storage space according to any one of the second to fourth embodiments or the exposure control system 50 for performing the rationalized exposure of the common storage space according to the first embodiment. Please refer to the first to fourth embodiments for a specific technical solution, which is not described herein again.
The above-listed detailed description is only a specific description of a possible embodiment of the present invention, and they are not intended to limit the scope of the present invention, and equivalent embodiments or modifications made without departing from the technical spirit of the present invention should be included in the scope of the present invention.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Furthermore, it should be understood that although the present description refers to embodiments, not every embodiment may contain only a single embodiment, and such description is for clarity only, and those skilled in the art should integrate the description, and the embodiments may be combined as appropriate to form other embodiments understood by those skilled in the art.
Claims (11)
1. A method of rationalizing exposure to a common storage space, the method comprising:
pre-constructing access conditions in the plug-in module;
the constraint module receives and analyzes the access request, calls the access condition in the plug-in module and only constrains the access request meeting the access condition;
the statistical module is used for counting and updating the resource occupation rate of the access request to the public storage space and submitting real-time statistical data representing the public storage space to a provider of the public storage space;
the provider of the public storage space periodically updates the access conditions in the plug-in modules based on the real-time statistics.
2. The method according to claim 1, wherein the plug-in module analyzes the Authorization field and the Host field of the HTTP request to count the access index generated by the access request sent by the user accessing the public storage space.
3. The method of claim 2, wherein the access metrics include a requestor attribute, a path of the access request, an address of a common memory space to which the access request is directed, and a data stream.
4. The method of claim 1, further comprising: and formatting the real-time statistical data, generating formatted data in an HTML format, and reporting the formatted data to a provider.
5. The method of claim 1, further comprising determining, by the constraint module, whether an access request to access the common storage space was submitted by a deployer;
when the deployment party submits the access request, the access request is not restricted;
when the user submits the access request, the access condition in the plug-in module is called, and only the access request meeting the access condition is restricted.
6. The method according to any one of claims 1 to 5, wherein the access condition includes access times, data traffic, access time, and user identity information to individual data storage spaces in a common storage space.
7. The method of claim 6, wherein the updating is in particular executing a nginx-s reload instruction, loading the latest LUA file.
8. The method of claim 1, further comprising:
counting the residual storage space in the public storage space and reporting to a provider;
the provider persists the residual storage space of the plurality of buckets in the common storage space to take the screened buckets which are not fully stored as candidate buckets;
and the file boot storage corresponding to the access request submitted by the user at the next time is put into the matched candidate storage bucket.
9. The method of claim 8, further comprising: dividing the file corresponding to the access request submitted next time by the user according to a set division strategy;
the segmentation strategy is as follows: the size of one of the file fragments formed through the division is matched with the residual storage space of the candidate buckets in the common storage space.
10. An exposure control system for rationalizing exposure of a common storage space, the exposure control system comprising:
a statistic module composed of a statistic request monitoring module, a data processing module and a data receiving module,
a plug-in module composed of a request constraint module and an index statistic module,
the constraint module consists of a constraint request monitoring module, a constraint judging module and a constraint executing module; wherein,
the constraint module is used for receiving and analyzing the access request, calling the access condition in the plug-in module and only constraining the access request meeting the access condition,
the statistical module is used for counting and updating the resource occupation rate of the access request to the public storage space and submitting real-time statistical data representing the public storage space to a provider of the public storage space,
the plug-in module is used for storing the access conditions and receiving updates of the access conditions in the plug-in module by a provider of the public storage space periodically according to the real-time statistical data.
11. A cloud storage system, comprising:
a number of storage components located in different geographical areas,
a logical processing cluster connected to the storage component,
the reverse-proxy cluster, and,
an exposure control system connecting the logic processing cluster and the reverse proxy cluster;
the exposure control system is configured as a system to perform the method of any of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710067189.XA CN106775499B (en) | 2017-02-07 | 2017-02-07 | A kind of pair of public memory space carries out rationalizing the method for exposure, exposure control system and a kind of cloud storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710067189.XA CN106775499B (en) | 2017-02-07 | 2017-02-07 | A kind of pair of public memory space carries out rationalizing the method for exposure, exposure control system and a kind of cloud storage system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106775499A true CN106775499A (en) | 2017-05-31 |
| CN106775499B CN106775499B (en) | 2019-02-05 |
Family
ID=58956149
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710067189.XA Active CN106775499B (en) | 2017-02-07 | 2017-02-07 | A kind of pair of public memory space carries out rationalizing the method for exposure, exposure control system and a kind of cloud storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106775499B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113411398A (en) * | 2021-06-18 | 2021-09-17 | 全方位智能科技(南京)有限公司 | Big data-based file cleaning writing and cleaning management system and method |
| CN114221994A (en) * | 2021-12-15 | 2022-03-22 | 北京安盟信息技术股份有限公司 | Dynamic allocation method for PCIE (peripheral component interface express) password card virtualized resources |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102508902A (en) * | 2011-11-08 | 2012-06-20 | 西安电子科技大学 | Block size variable data blocking method for cloud storage system |
| CN102780759A (en) * | 2012-06-13 | 2012-11-14 | 合肥工业大学 | Cloud computing resource scheduling method based on scheduling object space |
| CN102882983A (en) * | 2012-10-22 | 2013-01-16 | 南京云创存储科技有限公司 | Rapid data memory method for improving concurrent visiting performance in cloud memory system |
| CN102915374A (en) * | 2012-11-07 | 2013-02-06 | 北京搜狐新媒体信息技术有限公司 | Method, device and system for controlling access to database resources |
| CN102932279A (en) * | 2012-10-30 | 2013-02-13 | 北京邮电大学 | Multidimensional resource scheduling system and method for cloud environment data center |
| CN102968423A (en) * | 2012-03-27 | 2013-03-13 | 广州市国迈科技有限公司 | Design of high-performance private cloud storage node file system on basis of data container |
| CN103167021A (en) * | 2013-02-01 | 2013-06-19 | 浪潮(北京)电子信息产业有限公司 | Resource allocation method and resource allocation device |
| CN103729236A (en) * | 2013-12-29 | 2014-04-16 | 国云科技股份有限公司 | Method for limiting resource using limit of cloud computing users |
| CN104679905A (en) * | 2015-03-20 | 2015-06-03 | 四川特伦特科技股份有限公司 | High-speed storage system based on cloud storage |
-
2017
- 2017-02-07 CN CN201710067189.XA patent/CN106775499B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102508902A (en) * | 2011-11-08 | 2012-06-20 | 西安电子科技大学 | Block size variable data blocking method for cloud storage system |
| CN102968423A (en) * | 2012-03-27 | 2013-03-13 | 广州市国迈科技有限公司 | Design of high-performance private cloud storage node file system on basis of data container |
| CN102780759A (en) * | 2012-06-13 | 2012-11-14 | 合肥工业大学 | Cloud computing resource scheduling method based on scheduling object space |
| CN102882983A (en) * | 2012-10-22 | 2013-01-16 | 南京云创存储科技有限公司 | Rapid data memory method for improving concurrent visiting performance in cloud memory system |
| CN102932279A (en) * | 2012-10-30 | 2013-02-13 | 北京邮电大学 | Multidimensional resource scheduling system and method for cloud environment data center |
| CN102915374A (en) * | 2012-11-07 | 2013-02-06 | 北京搜狐新媒体信息技术有限公司 | Method, device and system for controlling access to database resources |
| CN103167021A (en) * | 2013-02-01 | 2013-06-19 | 浪潮(北京)电子信息产业有限公司 | Resource allocation method and resource allocation device |
| CN103729236A (en) * | 2013-12-29 | 2014-04-16 | 国云科技股份有限公司 | Method for limiting resource using limit of cloud computing users |
| CN104679905A (en) * | 2015-03-20 | 2015-06-03 | 四川特伦特科技股份有限公司 | High-speed storage system based on cloud storage |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113411398A (en) * | 2021-06-18 | 2021-09-17 | 全方位智能科技(南京)有限公司 | Big data-based file cleaning writing and cleaning management system and method |
| CN114221994A (en) * | 2021-12-15 | 2022-03-22 | 北京安盟信息技术股份有限公司 | Dynamic allocation method for PCIE (peripheral component interface express) password card virtualized resources |
| CN114221994B (en) * | 2021-12-15 | 2022-09-13 | 北京安盟信息技术股份有限公司 | Dynamic allocation method for PCIE (peripheral component interface express) password card virtualized resources |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106775499B (en) | 2019-02-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9497139B2 (en) | Client-allocatable bandwidth pools | |
| US11334929B2 (en) | Managing resource requests that exceed reserved resource capacity | |
| US9800465B2 (en) | Application placement through multiple allocation domain agents and flexible cloud scheduler framework | |
| US11252220B2 (en) | Distributed code execution involving a serverless computing infrastructure | |
| US9491313B2 (en) | Optimizing storage between mobile devices and cloud storage providers | |
| US9154589B1 (en) | Bandwidth-optimized cloud resource placement service | |
| US10826799B2 (en) | Apparatus for providing cloud service based on cloud service brokerage and method thereof | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| US20190081793A1 (en) | Parallel-chain architecture for blockchain systems | |
| US9306870B1 (en) | Emulating circuit switching in cloud networking environments | |
| US8918439B2 (en) | Data lifecycle management within a cloud computing environment | |
| US9037505B2 (en) | Mechanisms for executing a process in a cloud computing environment | |
| US10616139B1 (en) | Reducing quota access | |
| US20020194251A1 (en) | Systems and methods for resource usage accounting in information management environments | |
| US11064041B2 (en) | Apparatus for providing cloud service using cloud service brokerage based on multiple clouds and method thereof | |
| JP2016514311A (en) | Database system providing single tenant and multi-tenant environments | |
| US11687383B1 (en) | Distributed API accounting | |
| US10846788B1 (en) | Resource group traffic rate service | |
| CN110191160A (en) | A kind of concurrency control method and device | |
| CN114553960A (en) | Data caching method, device, equipment and storage medium | |
| CN106775499B (en) | A kind of pair of public memory space carries out rationalizing the method for exposure, exposure control system and a kind of cloud storage system | |
| US11558385B2 (en) | Automatic transaction processing failover | |
| WO2023093194A1 (en) | Cloud monitoring method and cloud management platform | |
| KR102637173B1 (en) | System of Blockchain Framework for 5G Mobile Edge Computing, Device and Management Method thereof | |
| US11556383B2 (en) | System and method for appraising resource configuration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 214125 Wuxi science and Technology Park, Jiangsu Binhu District No. 6 Patentee after: Huayun data holding group Co., Ltd Address before: No.6, science and education software park, Binhu District, Wuxi City, Jiangsu Province Patentee before: WUXI CHINAC DATA TECHNICAL SERVICE Co.,Ltd. |