CN104754001A - Cloud storage system and data storage method - Google Patents
Cloud storage system and data storage method Download PDFInfo
- Publication number
- CN104754001A CN104754001A CN201310745094.0A CN201310745094A CN104754001A CN 104754001 A CN104754001 A CN 104754001A CN 201310745094 A CN201310745094 A CN 201310745094A CN 104754001 A CN104754001 A CN 104754001A
- Authority
- CN
- China
- Prior art keywords
- stored
- storage
- server
- storage system
- cloud storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a cloud storage system and a data storage method. The cloud storage system comprises a storage server node, the storage server node comprises an object acquiring unit for acquiring an object to be stored, an object copying unit used for copying the object to be stored to obtain a plurality of object copies with the object to be stored, and a storage processing unit used for uniformly storing the object copies in different data fields in the cloud storage system. By means of the technical scheme, redundancy can be built through a standard server, and accordingly the data loss caused by single point failure is avoided, the cloud storage data safety can be improved, the position of each server in the cloud storage system is equal, the maintenance is facilitated, a storage server can be conveniently added to the system or deleted from the system, and the storage space is extensible.
Description
Technical field
The present invention relates to technical field of data storage, in particular to a kind of cloud storage system and a kind of date storage method.
Background technology
Cloud storage system, compared with heritage storage system, has following difference: the first, from functional requirement, cloud storage system towards polytype network online storage service, heritage storage system then towards such as high-performance calculation, the application such as transaction; The second, from performance requirement, first cloud storage system is wanted it is considered that the safety of data, reliably, the indexs such as efficiency, and due to userbase large, service range is wide, the features such as network environment is complicated and changeable, and realizing high-quality cloud stores service will face very large technological challenge; 3rd, from data management, cloud storage system not only will provide traditional file to access, and also wants can support Mass Data Management and provide public service support function, safeguards to facilitate cloud storage system back-end data.
Therefore, how to ensure the fail safe of cloud storage system data, the convenience of maintenance, the simplification of system and extensibility become technical problem urgently to be resolved hurrily at present.
Summary of the invention
The present invention is just based on the problems referred to above, propose a kind of OO cloud storage solution newly, standardized server is used to create redundancy, thus avoid the loss of data that Single Point of Faliure causes, can improve the fail safe that cloud stores data, each server position in cloud storage system is equal, easy to maintenance, and can add/delete storage server easily in systems in which, memory space easily extensible.
In view of this, the present invention proposes a kind of cloud storage system, comprise storage server node, described storage server node comprises: object acquisition unit, for obtaining object to be stored; Object tools unit, for copying described object to be stored, obtains the multiple object copies comprising described object to be stored; Storage processing unit, for being stored to the different pieces of information territory in described cloud storage system equably by described multiple object copies.
In this technical scheme, by copying object to be stored, and the multiple object copies obtained are evenly distributed in the different data field of storage area, realize the redundancy backup of object, time being broken down in individual data territory in storage server node, still provide service by other data fields, thus avoid the loss of data that Single Point of Faliure causes, contribute to improving the fail safe of data and the stability of cloud storage system.
Wherein, the process obtaining object to be stored can be that user asks to upload, and also can be the schedule backup to data such as virtual machine image, photo, Emails.Meanwhile, data field to choose very flexible, can be certain equipment, server, frame or even data center, can arrange according to the actual requirements.
According to a further aspect in the invention, additionally provide a kind of date storage method, for cloud storage system, described date storage method comprises: obtain object to be stored; Copy described object to be stored, obtain the multiple object copies comprising described object to be stored; Described multiple object copies is stored to equably the different pieces of information territory in described cloud storage system.
In this technical scheme, by copying object to be stored, and the multiple object copies obtained are evenly distributed in the different data field of storage area, realize the redundancy backup of object, time being broken down in individual data territory in storage server node, still provide service by other data fields, thus avoid the loss of data that Single Point of Faliure causes, contribute to improving the fail safe of data and the stability of cloud storage system.
Wherein, the process obtaining object to be stored can be that user asks to upload, and also can be the schedule backup to data such as virtual machine image, photo, Emails.Meanwhile, data field to choose very flexible, can be certain equipment, server, frame or even data center, can arrange according to the actual requirements.
By above technical scheme, standardized server can be used to create redundancy, thus avoid the loss of data that Single Point of Faliure causes, the fail safe that cloud stores data can be improved, each server position in cloud storage system is equal, easy to maintenance, and can add/delete storage server easily in systems in which, memory space easily extensible.
Those skilled in the art should understand, embodiments of the invention can be provided as method, system or computer program.Therefore, the present invention can adopt the form of complete hardware embodiment, completely software implementation or the embodiment in conjunction with software and hardware aspect.And the present invention can adopt in one or more form wherein including the upper computer program implemented of computer-usable storage medium (including but not limited to magnetic disc store, CD-ROM, optical memory etc.) of computer usable program code.
The present invention describes with reference to according to the flow chart of the method for the embodiment of the present invention, equipment (system) and computer program and/or block diagram.Should understand can by the combination of the flow process in each flow process in computer program instructions realization flow figure and/or block diagram and/or square frame and flow chart and/or block diagram and/or square frame.These computer program instructions can being provided to the processor of all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing device to produce a machine, making the instruction performed by the processor of computer or other programmable data processing device produce device for realizing the function of specifying in flow chart flow process or multiple flow process and/or block diagram square frame or multiple square frame.
These computer program instructions also can be stored in can in the computer-readable memory that works in a specific way of vectoring computer or other programmable data processing device, the instruction making to be stored in this computer-readable memory produces the manufacture comprising command device, and this command device realizes the function of specifying in flow chart flow process or multiple flow process and/or block diagram square frame or multiple square frame.
These computer program instructions also can be loaded in computer or other programmable data processing device, make on computer or other programmable devices, to perform sequence of operations step to produce computer implemented process, thus the instruction performed on computer or other programmable devices is provided for the step realizing the function of specifying in flow chart flow process or multiple flow process and/or block diagram square frame or multiple square frame.
Accompanying drawing explanation
Fig. 1 shows the schematic block diagram of cloud storage system according to an embodiment of the invention;
Fig. 2 shows the structural representation of cloud storage system according to an embodiment of the invention;
Fig. 3 shows the structural representation of storage server node according to an embodiment of the invention;
Fig. 4 shows the structural representation of Hash annular space according to an embodiment of the invention;
Fig. 5 shows the schematic diagram of the data mobile of consistency hash algorithm according to an embodiment of the invention;
Fig. 6 shows the schematic diagram of consistency hash algorithm constructing virtual node according to an embodiment of the invention;
Fig. 7 shows the schematic diagram of the corresponding relation between dummy node according to an embodiment of the invention and storage server node;
Fig. 8 shows the schematic flow diagram of date storage method according to an embodiment of the invention.
Embodiment
In order to more clearly understand above-mentioned purpose of the present invention, feature and advantage, below in conjunction with the drawings and specific embodiments, the present invention is further described in detail.It should be noted that, when not conflicting, the feature in the embodiment of the application and embodiment can combine mutually.
Set forth a lot of detail in the following description so that fully understand the present invention, but the present invention can also adopt other to be different from other modes described here and implement, and therefore, the present invention is not limited to the restriction of following public specific embodiment.
Fig. 1 shows the block diagram of cloud storage system according to an embodiment of the invention.
As shown in Figure 1, cloud storage system 100 according to an embodiment of the invention, comprises storage server node 102, and described storage server node 102 comprises: object acquisition unit 1022, for obtaining object to be stored; Object tools unit 1024, for copying described object to be stored, obtains the multiple object copies comprising described object to be stored; Storage processing unit 1026, for being stored to the different pieces of information territory in described cloud storage system 100 equably by described multiple object copies.
In this technical scheme, by copying object to be stored, and the multiple object copies obtained are evenly distributed in the different data field of storage area, realize the redundancy backup of object, time being broken down in individual data territory in storage server node 102, still provide service by other data fields, thus avoid the loss of data that Single Point of Faliure causes, contribute to improving the fail safe of data and the stability of cloud storage system.
Wherein, the process obtaining object to be stored can be that user asks to upload, and also can be the schedule backup to data such as virtual machine image, photo, Emails.
As one comparatively preferably embodiment, described storage processing unit 1026 is for performing storage operation by hash algorithm to described multiple object copies.
In this technical scheme, utilize the characteristic of hash algorithm, being uniformly distributed multiple object copies can be realized, thus make more to balance between each data field in cloud storage system, avoid Single Point of Faliure.Further, the preferred consistency hash algorithm of technique scheme, time as less in node to solve, what cannot realize multiple object copies is uniformly distributed problem.
As one comparatively preferably embodiment, described storage processing unit 1026 also for: according to the amount of capacity of each memory node in described storage server node 102, configure the actual of each memory node and take up room.
In this technical scheme, actually introduce the concept of " weight ", the ratio namely accounting for total memory capacity with the amount of capacity of each memory node represents.Because the memory node amount of capacity in cloud storage system is not quite similar, then by configuring larger weight for storage server capacious, the weight that the server configuration that capacity is little is less, thus system is when distributing the storage area of object to be stored, more object to be stored should be distributed to storage server capacious, distribute less object to be stored to the little storage server of capacity, to realize the equilibrium in performance, contribute to the runnability and the reliability that promote cloud storage system.
In the particular embodiment, such as: the amount of capacity of existing A and B two storage servers is respectively 80G and 20G, then total memory capacity size is 100G, the weight of A server is 80%(80/100), the weight of B server is 20%(20/100), if the file having user to upload 40G existing needs to store, then should store 32G(40G × 80% in A server) file, and B server should store 8G(40G × 20%) file.
As one comparatively preferably embodiment, described storage server node 102 also comprises: map and set up unit 1028, for obtaining the mark of described object to be stored, and the memory location that described multiple object copies is corresponding respectively, to set up the mapping relations between the mark of described object to be stored and all memory locations; And described cloud storage system 100 also comprises: proxy server nodes 104, described proxy server nodes 104 comprises: interface providing unit 1042, for externally providing service interface; Object searches unit 1044, during for receiving the operation requests to described object to be stored at described service interface, according to mark and the described mapping relations of described object to be stored, searches and obtains described object to be stored.
In this technical scheme, mapping relations are set up at described storage object and between copy and its corresponding stored position by " Hash " algorithm, not only can ensure that storage object and copy are placed in different territories uniformly, and when receiving the operation requests to storage object, just can inquire about according to the mark of storage object and mapping relations and obtaining storage object.
Wherein, the method utilizing " Hash " algorithm to set up mapping can be: (1) checks whether the title of storage object conforms with the regulations.(2) calculate cryptographic Hash corresponding to storage object title by " Hash ", in annular hash space, determine the memory location of this storage object.(3) described storage object and copy are by setting up mapping relations between cryptographic Hash and its corresponding stored position.
It should be noted that: not only said method can be adopted to store to the storage of multiple object copies, the method be mapped on Hash ring can be taked equally to carry out storage administration for the account mentioned in technique scheme, container.
It will be understood by those skilled in the art that and also exist a lot for storage object and the mode of setting up mapping between copy and its memory location, technique scheme is also not used in concrete restriction.Such as at some in other technological means, user can also come memory allocated position by the IP address of storage object.
As one comparatively preferably embodiment, described cloud storage system 100 also comprises: authentication server node 106, described authentication server 106 comprises: information receiving unit 1062, for receiving the subscriber identity information corresponding to described operation requests from described proxy server nodes 104; Identification authenticating unit 1064, for carrying out certification to described subscriber identity information, and returns by authentication result to described proxy server nodes 104, to be determined whether in response to described operation requests by described proxy server nodes 104.
In this technical scheme, limited memory space, as a kind of resource, can be set with different authorities to different users.When user sends operation requests, first according to the identity information of user, determine its authority had, according to authority with or without determining whether operation response request.
Specifically, user can be divided into domestic consumer and paying customer, and there is the memory space of 1G in domestic consumer, can realize uploading, downloading, the function that storage object is regularly preserved; Paying customer has the space of 10G or more, can realize uploading, download, the function such as real time inspection, storage object persistence.When user clicks storage object request real time inspection file content, then system need carry out certification to the identity information of user, if be domestic consumer, then without real time inspection authority, prompting user does not have real time inspection authority; If be paying customer, then there is real time inspection authority, the real time inspection request of user can be responded.
The structural representation of this cloud storage system 100 is shown corresponding to the cloud storage system 100, Fig. 2 shown in Fig. 1.
As shown in Figure 2, each functional module in the cloud storage system 100 shown in composition graphs 1, each server node in cloud storage system 100 is respectively used to realize:
Proxy server nodes (Proxy Node) 104, mainly bears the function of the meta data server of object storage system, provides the ability that data are located.Proxy server nodes 104 in this programme, be responsible for safeguarding account (account), container (container) and the positional information of object (object) in ring (ring), externally provide API service, process user access request simultaneously.In addition, the node scheduling work of coordinate storage server cluster is responsible for by proxy server, when a certain or multiple stage storage server node 102 cannot normally work, proxy server nodes 104 will the API request forward of user to the storage server node 102 of other normal operations, realize user without break in service ability.
Storage server node (Storage Node) 102, as object storage device, mainly realizes the memory function of user object data.Storage server node 102 is provided in the file object stores service of ability aspect not second to the special purpose memory devices of costliness based on the hardware server equipment of the universal standard, in cost, have clear superiority.Storage server node 102 runs service processes and other associated process of user account, container and object.
Identity authentication certificate server node (Auth Node) 106, user identity is provided to identify authentication service, proxy server nodes 104 is when processing user and asking, need to carry out qualification certification to the identity of client, the identifying data information that now user provides by proxy server nodes 104 is forwarded to qualification certificate server node 106 and processes.
Wherein, related to the concepts such as account (account), container (container), object (object) and ring (ring) above, be described in detail below.
1, account
This programme purpose of design is for multi-user provides object stores service, each user has segregated account, user needed to submit account name and key to before use object stores service, and proxy server nodes 104 receives and the information that user provides is submitted to identity authentication Verification System.Identity authentication Verification System is safeguarded by identity authentication certificate server node 106, for the identification of the validity of account, and result is returned to proxy server nodes 104.According to result, proxy server nodes 104 judges that user can use this programme object stores service to carry out data processing.
2, container
Container is equivalent to the file under Windows operating system or the catalogue under unix system, for organising data, is independently between memory block.But container and file or the most basic difference of catalogue are that container can not be nested, and namely container can not have subordinate's container to exist.In the account of this programme, the quantity of container does not have the upper limit, but this programme requires that object (file) must be stored in the middle of certain container, and therefore minimum in the account should have a container to provide uploading of object (file).
The name of container does not have too large restriction, and this programme only will go to comprise slash symbols in Container Name and Name Length must be less than 256 bytes.Wherein, the length of Container Name refers to that Container Name converts the length of URL coding to, and if Container Name is " Course Docs ", this title converts to " Course%20Docs " after URL coding, thus the length of Container Name is 13 bytes, instead of 11 bytes in the imagination.
3, object
Object is made up of storage entity and metadata, is equivalent to the concept of file.When to this programme object storage system (being safeguarded by storage server node 102) upload file, file is not through compression or encryption, but and file deposit place (container name), object name and file the information composition object of metadata (key-value pair form), then deposit on storage server node 102.
In addition, object naming and container are named similar, there is no too much restriction, and this programme only requires that object oriented length must be less than 256 bytes.Similar containers Name Length, object oriented length refers to the length of the character string that object oriented becomes through URL code conversion.Such as, object oriented is " C++final (v2) .txt ", becomes character string " C%2B%2Bfinal%28v2%29.txt " through URL code conversion, and this object oriented length is 24 bytes, instead of 16 bytes.In addition, it is 0 byte that this programme object storage system allows the object uploaded minimum, is 5GB to the maximum, but can carry out segmentation by the large objects support function of this programme to the object more than 5GB and upload.This programme also limits to some extent for the metadata in object, and the source data of each object is not deposited more than 90 key-value pairs, and total key-value pair take up room can not more than 4KB.
4, ring
Ring represents the mapping relations of storage entity title and storage entity physical location, for determining the particular location of deposit data in storage server cluster.In the object storage system of this programme, account, container and object can have independently ring, but the operation principle of each ring is the same.When user is from external call API Access this programme system, proxy server nodes 104 can find corresponding ring, to determine the physical location at the entity place that will operate according to operated entity (account/containers/objects).
(1) ring employs territory (zone), equipment (device), and the concept of subregion (partition) and copy (replicas) carrys out the mapping relations of maintenance entity and physical location.
First, for " territory " and " equipment ", Fig. 3 shows the structural representation of storage server node according to an embodiment of the invention.Wherein, storage server node 102 specifically can be divided into multiple data field, isolates for data." territory " (i.e. a data field) can be made up of one " equipment "; Meanwhile, because the definition in territory is very flexible, also can be the equipment of some quantity, server, frame or even data center.By the division to multiple data field, under can ensureing the scene lost efficacy in certain territory, this programme system continues to provide service by other active domains.
Secondly, in whole storage server cluster, each subregion default copy 3 parts of ring is (for citing; Obviously also can be other numbers), i.e. 3 copies (comprising body); Wherein, the particular location of partitioned storage is also safeguarded by ring.The subregion of ring is evenly distributed on all devices of whole this programme object storage system.When adding when there being equipment or delete, ring ensure that the partition data of each movement is minimum as far as possible, and only comprises a copy of subregion in the subregion of each movement.
In fact, the dummy node of concept source in " Hash (Hash) " algorithm of subregion, this algorithm is mainly used in solving the hot issue in the Internet, when namely increasing newly or delete the server in cluster, the link of user should be the least possible move on other servers, system can not be much affected because increase and decrease server.This programme applies hash algorithm equally, and when the design of ring is for newly-increased or deletion memory device, therefore this programme system can not be subject to too much impact.
(2) weight (weight) is also the key concept in ring.When the memory capacity of the storage server node 102 of this programme system is not of uniform size, ensure being uniformly distributed of subregion by weight parameter.More subregion is responsible for by the memory device that capacity is larger, and the memory device that capacity is less only needs responsible less subregion, and the subregion that therefore capacity is larger has larger weight, and the less subregion weight of capacity is also less.
(3) particularly, the present invention employs consistency hash algorithm in ring.The object of consistency hash algorithm is mainly in order to solve the hot issue in distributed network.In general, 4 indexs of a hash algorithm are weighed:
1) balance (Balance)
Balance refers to that the result of Hash can be evenly distributed as far as possible, makes full use of all spatial caches.
2) monotonicity (Monotonicity)
Monotonicity is referred to if there have been some contents to be assigned in corresponding buffering by hash algorithm, has again new buffering to join in system.The result of Hash should be able to ensure that original content of having distributed can be mapped in new buffering and go, and can not be mapped to other buffering areas in old buffered collection.
3) dispersed (Spread)
Dispersiveness defines in distributed environment, different terminals by Hash process by content map to buffer memory time, because visible buffer is different, Hash result is inconsistent, and identical content is mapped to different buffering areas.
4) load (Load)
Load is another latitude required dispersiveness.Since different terminals by identical content map in different buffering areas, so for a specific buffering area, also can may be mapped as different contents by different users.
This programme uses the main purpose of hash algorithm to be (increase/delete server) when changing the node quantity of cluster in the design of changing, and can change the mapping relations of already present key (key) and node (node) as few as possible, to meet monotonicity.So, for the embody rule thinking of consistency hash algorithm, be described in detail below in conjunction with Fig. 4 to Fig. 7.Wherein, Fig. 4 shows the structural representation of Hash annular space according to an embodiment of the invention; Fig. 5 shows the schematic diagram of the data mobile of consistency hash algorithm according to an embodiment of the invention; Fig. 6 shows the schematic diagram of consistency hash algorithm constructing virtual node according to an embodiment of the invention; Fig. 7 shows the schematic diagram of the corresponding relation between dummy node according to an embodiment of the invention and storage server node.
As shown in Figure 4, in Hash annular space structure according to an embodiment of the invention, storage server 1, storage server 2 etc. are when storage object, and the step of employing comprises:
A, first, obtain the cryptographic Hash of each storage server node (machine name or IP address), and be assigned to upper (getting 0-232 here) between an annular regions as shown in Figure 4.
B, then, obtain the cryptographic Hash needing storage object, and be assigned on above-mentioned annulus according to this cryptographic Hash.
C, last, from object map to position search clockwise, and object to be saved on first storage server node finding.
Based on above-mentioned storage mode, then for the increase and decrease of storage server node, will greatly reduce the migration amount to data.Ratio as shown in Figure 5, suppose in the hash space of annular, storage server 5 is mapped between storage server 3 and storage server 4, so affected will only: travel through counterclockwise along storage server 5, until the object (these objects are mapped on storage server 4 originally) between next storage server (i.e. storage server 3).
Consider that hash algorithm is not ensure absolute balance, especially node is less, and object can not be mapped on node uniformly.In order to solve this situation, consistency Hash introduces the concept of " dummy node ".As shown in Figure 6, " dummy node " (circular node larger in figure) is the duplicate of actual node at annular space, and actual node is corresponding several " dummy nodes ", and " dummy node " arranges with cryptographic Hash in hash space.
Particularly, Fig. 7 shows the corresponding relation between dummy node under a kind of specific embodiment and actual node.As described in Figure 7, dummy node comprises V1, V2 ... V8, and actual node comprises C1, C2 ... C4.Object 1, object 2 etc. are after the computing of hash algorithm, be assigned to corresponding dummy node, and utilize the corresponding relation between dummy node and actual node, corresponding relation etc. such as between V1, V2 and C1, then finally object 1, object 2 etc. can be stored in corresponding actual node, and ensure that when the negligible amounts of actual node, still can realize uniform object distribution.
Fig. 8 shows the schematic flow diagram of date storage method according to an embodiment of the invention.
As shown in Figure 8, date storage method according to an embodiment of the invention, for cloud storage system, described date storage method comprises:
Step 802, obtains object to be stored;
Step 804, copies described object to be stored, obtains the multiple object copies comprising described object to be stored;
Step 806, is stored to the different pieces of information territory in described cloud storage system equably by described multiple object copies.
In this technical scheme, by copying object to be stored, and the multiple object copies obtained are evenly distributed in the different data field of storage area, realize the redundancy backup of object, time being broken down in individual data territory in storage server node, still provide service by other data fields, thus avoid the loss of data that Single Point of Faliure causes, contribute to improving the fail safe of data and the stability of cloud storage system.
Wherein, the process obtaining object to be stored can be that user asks to upload, and also can be the schedule backup to data such as virtual machine image, photo, Emails.Meanwhile, data field to choose very flexible, can be certain equipment, server, frame or even data center, can arrange according to the actual requirements.
As one comparatively preferably embodiment, by hash algorithm, storage operation is performed to described multiple object copies.
In this technical scheme, utilize the characteristic of hash algorithm, being uniformly distributed multiple object copies can be realized, thus make more to balance between each data field in cloud storage system, avoid Single Point of Faliure.Further, the preferred consistency hash algorithm of technique scheme, time as less in node to solve, what cannot realize multiple object copies is uniformly distributed problem.
As one comparatively preferably embodiment, according to the amount of capacity of each memory node in described cloud storage system, configure the actual of each memory node and take up room.
In this technical scheme, actually introduce the concept of " weight ", the ratio namely accounting for total memory capacity with the amount of capacity of each memory node represents.Because the memory node amount of capacity in cloud storage system is not quite similar, then by configuring larger weight for storage server capacious, the weight that the server configuration that capacity is little is less, thus system is when distributing the storage area of object to be stored, more object to be stored should be distributed to storage server capacious, distribute less object to be stored to the little storage server of capacity, to realize the equilibrium in performance, contribute to the runnability and the reliability that promote cloud storage system.
In the particular embodiment, such as: the amount of capacity of existing A and B two storage servers is respectively 80G and 20G, namely total memory capacity size is 100G, then the weight of A server can be 80%(80/100), the weight of B server can be 20%(20/100).If have the file of 40G need store, then according to above-mentioned weight allocation plan, in A server, should 32G(40G × 80% be stored) file, and B server should store 8G(40G × 20%) file.
As one comparatively preferably embodiment, obtain the mark of described object to be stored, and the memory location that described multiple object copies is corresponding respectively, to set up the mapping relations between the mark of described object to be stored and all memory locations; When receiving the operation requests to described object to be stored, according to mark and the described mapping relations of described object to be stored, searching and obtaining described object to be stored.
In this technical scheme, mapping relations are set up at described storage object and between copy and its corresponding stored position by " Hash " algorithm, not only can ensure that storage object and copy are placed in different territories uniformly, and when receiving the operation requests to storage object, just can inquire about according to the mark of storage object and mapping relations and obtaining storage object.
Wherein, the method utilizing " Hash " algorithm to set up mapping can be: (1) checks whether the title of storage object conforms with the regulations.(2) calculate cryptographic Hash corresponding to storage object title by " Hash ", in annular hash space, determine the memory location of this storage object.(3) described storage object and copy are by setting up mapping relations between cryptographic Hash and its corresponding stored position.
It should be noted that: not only said method can be adopted to store to the storage of multiple object copies, the method be mapped on Hash ring can be taked equally to carry out storage administration for the account mentioned in technique scheme, container.
It will be understood by those skilled in the art that and also exist a lot for storage object and the mode of setting up mapping between copy and its memory location, technique scheme is also not used in concrete restriction.Such as at some in other technological means, user can also come memory allocated position by the IP address of storage object.
As one comparatively preferably embodiment, certification is carried out to the subscriber identity information sending described operation requests; And determine whether in response to described operation requests according to authentication result.
In this technical scheme, limited memory space, as a kind of resource, can be set with different authorities to different users.When user sends operation requests, first according to the identity information of user, determine its authority had, according to authority with or without determining whether operation response request.
Specifically, user can be divided into domestic consumer and paying customer, and there is the memory space of 1G in domestic consumer, can realize uploading, downloading, the function that storage object is regularly preserved; Paying customer has the space of 10G or more, can realize uploading, download, the function such as real time inspection, storage object persistence.When user clicks storage object request real time inspection file content, then system need carry out certification to the identity information of user, if be domestic consumer, then without real time inspection authority, prompting user does not have real time inspection authority; If be paying customer, then there is real time inspection authority, the real time inspection request of user can be responded.
In order to realize above-mentioned functions, in cloud storage system 100 specific implementation process mentioned in technique scheme, controlled by the process run on proxy server nodes 104, storage server node 102, identity authentication certificate server node 106 and realized, be explained below in conjunction with the main steps run in three server nodes.
One, proxy server nodes 104
Proxy server process: proxy server process (" proxy-server ") is the process that this programme proxy service node 104 is run.Agency service process safeguards account, container and the object positional information in ring, and externally provides API service (API that namely external network can be provided by proxy server carries out the access of data), the request of process user API Access.
Wherein, " memcached " is high performance distributed memory target cache system, and it reduces the number of times of reading database by data cached in internal memory and object.This programme agency service process uses " memcached " to carry out the buffer memory of data, to improve the access speed of user.
Two, storage server node 102
Storage server process mainly comprises three classes: Account class process, Container class process and Object class process, as shown in table 1 below:
| Account | account-server,account-replicator,account-auditoraccount-reaper |
| Container | container-server,container-replicator,container-updater,container-auditor |
| Object | object-server,object-replicator,object-updater,object-auditor,object-expirer |
Table 1
1, object server process (object-server)
Object server process is responsible for the storage of data (object) at local device, the lifecycle operation such as acquisition and deletion.Object stores with the form of binary file, and the metadata of object will leave in the middle of the additional attribute (xattrs) of binary file.The path that object is deposited is the title of object adds operation timestamp through the result that Hash calculation draws, and the last write operation of object determines the content of object, while system external service, ensure up-to-date object version.Deleting object operation also can as the once renewal being object, and obj ect file will become 0 byte, and filename terminates with " .ts ", and wherein ts is the abbreviation of tombstone, looks like for gravestone, illustrates that this file is deleted.Such way guarantees that the object deleted can correctly be copied, and the object of legacy version can not be reappeared because of some failure scene.
2, container server process (container-server)
The process operation of container server process primary responsibility object, this process collects the information of container that each object is deposited, and be Sqlite database file by these information data storings, similar with obj ect file, this file can carry out operations such as copying in the cluster.In addition, container server process also can be added up container, comprises the sum of object in each container and the memory space etc. of container use.
3, account service process (account-server)
Account service process and container service process similar, difference be a series of container of account service process primary responsibility process instead of object.
4, data Replica process (replicator)
Data Replica copying and simultaneous operation mainly to data file.Wherein, the object copied is to provide system reliability, avoids Single Point of Faliure; And synchronous target ensures that file is in coherency state, make the file provided same family be all consistent, and be latest edition.This programme usage data duplicating process realizes the operation of data Replica, and the data that this locality will be stored and the long-range copy of corresponding every portion thereof are made comparisons, and guarantee that all data are all latest editions.The data Replica process of object adopts Hash table to carry out the sub-part of each subregion of rapid comparison; The data Replica process of container and account then adopts Hash table and digital watermark to carry out the rapid comparison of data, the renewal rewards theory of data Replica is based on Push technology, for object data duplicating process, renewal rewards theory is just simply the remote storage server of file synchronization to other; And for the duplicating process of account and container data, need, by http protocol, the record lacked is pushed to remote storage server, or synchronous whole Sqlite database file.
5, database data duplicating process
" account-replicator " and " container-replicator " all belongs to database replication process.The first step of database data duplicating process is the consistency using Hash calculation to verify two databases, confirm that whether the database in storage server cluster is synchronous, if the result of the database on storage server cluster after Hash calculation is inconsistent, so data Replica enters and can share up-to-date synchronous point record towards that, carry out the synchronous of database, synchronous point is stored in inside each database with the form of " Record ID number " and " No. ID, database " tuple.Wherein, database ID is unique, and namely No. ID, each database is unique in storage server cluster; Record ID number is then monotonically increasing integer.If the database on certain storage server disappears completely, so local database files is transferred on the storage server that database disappears by rsync synchronizing software unexpectedly, and the database ID after this on this storage server can be modified as new and be unique No. ID.
6, object data duplicating process (object-replicator)
The initial implementation of object data duplicating process is only use rsync synchronizing software to carry out the propelling movement of the data on local storage server to remote storage server.This implementation shows well in small-scale storage server cluster, but when low memory, duplicating process will consume the plenty of time.Current object data duplicating process comparatively before pattern improve to some extent, by calculating the cryptographic Hash of each directory content in each subregion, result is saved to Hash file, thus know the partition content after being modified, then inefficacy cryptographic Hash being sent to has in each remote storage server of corresponding subregion equally, and remote storage server carries out data syn-chronization for associative directory.The document backup of amendment after remote storage server, object tools process notify these synchronous after storage server recalculate the cryptographic Hash of corresponding document catalogue, and be saved in the Hash file of corresponding subregion and go.
7, process (-updater) is upgraded
" account-updater ", " container-updater ", " object-updater " process all belongs to renewal process.There is the situation that container or account data are not instantly updated in this programme operating environment, usually appear at server node and make mistakes or period of high capacity.If the renewal rewards theory failure of container or account, so these renewal rewards theory will line up queue in local service, are finally processed the renewal rewards theory of these failures by renewal process.Such as 1, while the server node at container place is in high load condition, need certain object to be added in the middle of this programme storage system, now proxy server is responded the object that adds of user and to be entered into force and readable, but server fails the list object that upgrades in time due to high capacity, that is, user object is not added under certain container of specifying, the renewal rewards theory of this failure will add in queue and can be processed by renewal process after a while.In the process that object data upgrades, relate to the concept of " consistency window ".In actual applications, consistency window is relevant with the running time of the process of renewal, and what it was specified is the time that All Files is in coherency state.When the process of renewal is run, illustrate that the renewal rewards theory of failure needs process, the obj ect file now in illustrative system has been in inconsistent state; When renewal process is not reruned, illustrate that all renewal rewards theory are disposed, the file in system is consistent.The time period that renewal process is not run, system is in coherency state, is called consistency window during this period of time.Consistency window perhaps can not sense by proxy server, reason is that proxy server can a series of request forward to first cell therefor server processes, the container server being in high capacity is not the process that uniquely can process request, so just can cause the situation that proxy server errored response user asks.
8, audit progress (-auditor)
" account-auditor ", " container-auditor ", " object-auditor " all belongs to audit progress.Audit progress operates on storage server, performs check object, the integrality of container and account.If there is auditing errors, data will be isolated, and simultaneously data Replica process will obtain copy corresponding to data from other storage servers and replaces the data made a mistake.
9, account removes process (account-reaper)
It is backstage finger daemon that account removes process, runs on the server at account service process (account-server) place, and primary responsibility removes the data comprised in deleted account.Account removes process scanning server once in a while, for finding the account database whether existing and be masked as deletion.
10, the expired process of object (object-expirer)
" object-expirer " process is provided in the service of deleting object in the fixed time.This programme user can use X-Delete-At or X-Delete-After Header to carry out the erasing time of appointed object in the HTTP request of PUT or POST method.When the RT of object expires, this programme group system automatic stopping can outwards provide the service of this object, and at short notice this object is deleted from system.Only an object-expirer process is needed, although even if the problem object-expirer process that there is Single Point of Faliure is out of service within a period of time, the normal operation of which business also can not be affected in a cluster.If in the process that object-expirer process is out of service, user accesses expired but not yet deleted object by the HTTP request of GET or HEAD method, so proxy server will return 404NOT FOUND mistake to user, after object-expirer process is restarted, these objects will be deleted very soon.Object-expirer backstage finger daemon passes through the rear end storage server of the proxy access cluster of therein, but need not access proxies or external network.
11, rsync synchronized process
Rsync is writing a Chinese character in simplified form of remote sync, means remote synchronization.Rsync is the data syn-chronization software of class UNIX operating system, can synchronized update file and catalogue, and suitably utilizes differential coding to reduce transfer of data.The stores service process intrinsic call rsync of this programme realizes the function of data syn-chronization.
Identity authentication certificate server process:
Identity authentication authentication service process is mainly used in providing this programme identity authentication authentication service, and its work mainly comprises following two kinds:
(1) identify: for verifying the validity of user in request.
(2) certification: for differentiating the authority that user resources are accessed.
At present, the identity authentication Verification System of this programme support has three kinds:
TempAuth is the embedded identity authentication Verification System of this programme, based on the existing identity authentication Verification System exploitation of Rackspace.TempAuth, with the form of WSGI middleware, can run as the subsystem of this programme or external system.This programme is by the validity of Auth Token in the request of TempAuth verification system validates, and Buffering Verifier result; The value of Auth Token changes in can not asking again, but has the term of validity, and expired Auth Token will lose efficacy.
Token can be specified by X-Auth-Token and X-Storage-TokenHeader in HTTP request.This programme calls identity authentication Verification System and sends the Auth Token that will verify, for effective Token, identity authentication Verification System responded with the overall effective time of Token, and this programme puts into buffer memory Token simultaneously, until after exceeding the term of validity, then Token is deleted.
TempAuth identity authentication Verification System contains keeper and non-administrative users.Administrator can carry out any operation to the account at place; Non-administrative users can only carry out the operation allowing to perform in the container that account is specified.
More than be described with reference to the accompanying drawings technical scheme of the present invention, considered in correlation technique, in server stores, there is Single Point of Faliure, safeguarded consuming time, poor expandability.The present invention proposes a kind of OO cloud storage solution newly, standardized server is used to create redundancy, thus avoid the loss of data that Single Point of Faliure causes, the fail safe that cloud stores data can be improved, each server position in cloud storage system is equal, easy to maintenance, and can add/delete storage server easily in systems in which, memory space easily extensible.
Can set up a self-defining storage solution of software very easily by this programme, be especially object storage system (Object-Based Storage System).Object storage system combines NAS(Network Access Server, network access server) and SAN(Storage AreaNetwork, centralized management high-speed storage network) advantage, the high speed simultaneously with SAN is directly accessed and the advantages such as the data sharing of NAS, provides the storage architecture of the data sharing of high reliability, professional platform independence and safety.This platform can be separated management from the infrastructure of bottom, make client can set up Mass storage pond on hardware, video request program can be widely used in, user network data store, books, data, each application direction of document preservation etc., pass in time, huge effect must be played.
Those skilled in the art should understand, embodiments of the invention can be provided as method, system or computer program.Therefore, the present invention can adopt the form of complete hardware embodiment, completely software implementation or the embodiment in conjunction with software and hardware aspect.And the present invention can adopt in one or more form wherein including the upper computer program implemented of computer-usable storage medium (including but not limited to magnetic disc store, CD-ROM, optical memory etc.) of computer usable program code.
The present invention describes with reference to according to the flow chart of the method for the embodiment of the present invention, equipment (system) and computer program and/or block diagram.Should understand can by the combination of the flow process in each flow process in computer program instructions realization flow figure and/or block diagram and/or square frame and flow chart and/or block diagram and/or square frame.These computer program instructions can being provided to the processor of all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing device to produce a machine, making the instruction performed by the processor of computer or other programmable data processing device produce device for realizing the function of specifying in flow chart flow process or multiple flow process and/or block diagram square frame or multiple square frame.
These computer program instructions also can be stored in can in the computer-readable memory that works in a specific way of vectoring computer or other programmable data processing device, the instruction making to be stored in this computer-readable memory produces the manufacture comprising command device, and this command device realizes the function of specifying in flow chart flow process or multiple flow process and/or block diagram square frame or multiple square frame.
These computer program instructions also can be loaded in computer or other programmable data processing device, make on computer or other programmable devices, to perform sequence of operations step to produce computer implemented process, thus the instruction performed on computer or other programmable devices is provided for the step realizing the function of specifying in flow chart flow process or multiple flow process and/or block diagram square frame or multiple square frame.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for a person skilled in the art, the present invention can have various modifications and variations.Within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. a cloud storage system, is characterized in that, comprises storage server node, and described storage server node comprises:
Object acquisition unit, for obtaining object to be stored;
Object tools unit, for copying described object to be stored, obtains the multiple object copies comprising described object to be stored;
Storage processing unit, for being stored to the different pieces of information territory in described cloud storage system equably by described multiple object copies.
2. cloud storage system according to claim 1, is characterized in that, described storage processing unit is used for performing storage operation by hash algorithm to described multiple object copies.
3. cloud storage system according to claim 1 and 2, is characterized in that, described storage processing unit also for:
According to the amount of capacity of each memory node in described storage server node, configure the actual of each memory node and take up room.
4. cloud storage system according to claim 1 and 2, is characterized in that, described storage server node also comprises:
Unit is set up in mapping, for obtaining the mark of described object to be stored, and the memory location that described multiple object copies is corresponding respectively, to set up the mapping relations between the mark of described object to be stored and all memory locations; And
Described cloud storage system also comprises: proxy server nodes, and described proxy server nodes comprises:
Interface providing unit, for externally providing service interface;
Object searches unit, during for receiving the operation requests to described object to be stored at described service interface, according to mark and the described mapping relations of described object to be stored, searches and obtains described object to be stored.
5. cloud storage system according to claim 4, is characterized in that, also comprises: authentication server node, and described authentication server comprises:
Information receiving unit, for receiving the subscriber identity information corresponding to described operation requests from described proxy server nodes;
Identification authenticating unit, for carrying out certification to described subscriber identity information, and returns authentication result to described proxy server nodes, to be determined whether in response to described operation requests by described proxy server nodes.
6. a date storage method, for cloud storage system, is characterized in that, described date storage method comprises:
Obtain object to be stored;
Copy described object to be stored, obtain the multiple object copies comprising described object to be stored;
Described multiple object copies is stored to equably the different pieces of information territory in described cloud storage system.
7. date storage method according to claim 6, is characterized in that, also comprises:
By hash algorithm, storage operation is performed to described multiple object copies.
8. the date storage method according to claim 6 or 7, is characterized in that, also comprises:
According to the amount of capacity of each memory node in described cloud storage system, configure the actual of each memory node and take up room.
9. the date storage method according to claim 6 or 7, is characterized in that, also comprises:
Obtain the mark of described object to be stored, and the memory location that described multiple object copies is corresponding respectively, to set up the mapping relations between the mark of described object to be stored and all memory locations;
When receiving the operation requests to described object to be stored, according to mark and the described mapping relations of described object to be stored, searching and obtaining described object to be stored.
10. date storage method according to claim 9, is characterized in that, also comprises:
Certification is carried out to the subscriber identity information sending described operation requests; And
Determine whether in response to described operation requests according to authentication result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310745094.0A CN104754001A (en) | 2013-12-30 | 2013-12-30 | Cloud storage system and data storage method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310745094.0A CN104754001A (en) | 2013-12-30 | 2013-12-30 | Cloud storage system and data storage method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104754001A true CN104754001A (en) | 2015-07-01 |
Family
ID=53593098
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310745094.0A Pending CN104754001A (en) | 2013-12-30 | 2013-12-30 | Cloud storage system and data storage method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104754001A (en) |
Cited By (36)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105159919A (en) * | 2015-07-24 | 2015-12-16 | 福建师范大学 | Data multi-copy correlation method and system |
| CN105610864A (en) * | 2016-02-04 | 2016-05-25 | 上海盛巨信息技术有限公司 | Cloud bridge system for connecting NAS storage to cloud storage |
| CN105847364A (en) * | 2016-03-28 | 2016-08-10 | 乐视控股（北京）有限公司 | Public cloud object storage method based on uniform domain name and public cloud object storage system based on uniform domain name |
| CN106202396A (en) * | 2016-07-08 | 2016-12-07 | 乐视控股（北京）有限公司 | Object storage method and object storage system |
| CN106202367A (en) * | 2016-07-07 | 2016-12-07 | 腾讯科技（深圳）有限公司 | A kind of processing method and processing device of object information |
| CN107018185A (en) * | 2017-03-28 | 2017-08-04 | 清华大学 | The synchronous method and device of cloud storage system |
| CN107154966A (en) * | 2017-04-24 | 2017-09-12 | 浙江宇视科技有限公司 | A kind of video data wiring method and device |
| CN107528719A (en) * | 2017-03-08 | 2017-12-29 | 深圳市泽云科技有限公司 | A kind of implementation method for lifting cloud storage system high availability |
| CN107634853A (en) * | 2017-08-28 | 2018-01-26 | 深圳市云舒网络技术有限公司 | A kind of customizable distributed caching method based on object storage |
| CN107911461A (en) * | 2017-11-24 | 2018-04-13 | 网宿科技股份有限公司 | Object processing method, storage server and cloud storage system in cloud storage system |
| CN108536795A (en) * | 2018-04-02 | 2018-09-14 | 深圳市华傲数据技术有限公司 | A kind of method, medium and equipment improving diagram data interactive efficiency |
| CN108833535A (en) * | 2018-06-14 | 2018-11-16 | 浙江远算云计算有限公司 | User data storage method based on the storage of cloud platform distributed block |
| CN109325025A (en) * | 2018-07-31 | 2019-02-12 | 佛山科学技术学院 | A kind of restructural integrated approach of multi-source heterogeneous platform |
| CN109362236A (en) * | 2018-08-31 | 2019-02-19 | 深圳大学 | Secure storage method of data, device, equipment and storage medium based on double clouds |
| CN109471870A (en) * | 2018-11-16 | 2019-03-15 | 北京金山云网络技术有限公司 | Method, apparatus, electronic equipment and the computer-readable medium that resource data is read |
| US10289502B2 (en) | 2016-03-07 | 2019-05-14 | International Business Machines Corporation | User-defined failure domains for software-defined storage systems |
| CN109933421A (en) * | 2017-12-18 | 2019-06-25 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of distributed data processing |
| CN109936594A (en) * | 2017-12-15 | 2019-06-25 | 上海欣能信息科技发展有限公司 | A kind of cloud storage system based on electric power data |
| CN110287169A (en) * | 2019-06-28 | 2019-09-27 | 四川长虹电器股份有限公司 | A kind of master data moving method between cloud application |
| CN110457325A (en) * | 2019-08-12 | 2019-11-15 | 北京百度网讯科技有限公司 | Method and apparatus for output information |
| CN110457263A (en) * | 2019-08-13 | 2019-11-15 | 北京首都在线科技股份有限公司 | A kind of date storage method and device |
| CN110557975A (en) * | 2018-04-02 | 2019-12-10 | 甲骨文国际公司 | Tenant data comparison for multi-tenant identity cloud services |
| CN111107109A (en) * | 2020-01-08 | 2020-05-05 | 世纪恒通科技股份有限公司 | Log-in-free technology based on token |
| CN111309260A (en) * | 2020-02-16 | 2020-06-19 | 西安奥卡云数据科技有限公司 | Data storage node selection method |
| US10698618B1 (en) | 2019-03-26 | 2020-06-30 | International Buisness Machines Corporation | User-defined failure domains for software-defined storage systems |
| CN111475117A (en) * | 2020-04-09 | 2020-07-31 | 南京信息职业技术学院 | Aggregation cloud storage QoS control method and system |
| CN111637891A (en) * | 2020-05-06 | 2020-09-08 | 北京小趣智品科技有限公司 | Toilet positioning method and system |
| CN112035402A (en) * | 2019-06-04 | 2020-12-04 | 顺丰科技有限公司 | File storage method and device and terminal equipment |
| WO2021151293A1 (en) * | 2020-08-27 | 2021-08-05 | 平安科技（深圳）有限公司 | Hash ring load balancing method and apparatus, and electronic device and storage medium |
| CN113228593A (en) * | 2019-11-14 | 2021-08-06 | 斯诺弗雷克公司 | Database integrated external certificateless scratch pad |
| WO2021189308A1 (en) * | 2020-03-25 | 2021-09-30 | Beijing Didi Infinity Technology And Development Co., Ltd. | Delete operation in object storage system using enhanced meta structure |
| CN114385086A (en) * | 2022-01-17 | 2022-04-22 | 杭州瑞网广通信息技术有限公司 | Data storage method and device based on cloud storage system |
| CN114785795A (en) * | 2022-04-02 | 2022-07-22 | 北京蓝海在线科技有限公司 | Distributed storage method, system, device, storage medium and information processing terminal |
| CN115408396A (en) * | 2022-09-02 | 2022-11-29 | 金蝶征信有限公司 | Business data storage method and device, computer equipment and storage medium |
| US11675640B2 (en) | 2019-10-29 | 2023-06-13 | Snowflake Inc. | External function invocation by a data system |
| US11675784B2 (en) | 2021-04-30 | 2023-06-13 | Snowflake Inc. | Configuring parallelism parameters for invocation of external table functions |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101552799A (en) * | 2008-04-04 | 2009-10-07 | 华为技术有限公司 | Media node fault-tolerance method and device |
| CN101964820A (en) * | 2010-10-08 | 2011-02-02 | 中兴通讯股份有限公司 | Method and system for keeping data consistency |
| US20110099146A1 (en) * | 2009-10-26 | 2011-04-28 | Mcalister Grant Alexander Macdonald | Monitoring of replicated data instances |
| CN102307221A (en) * | 2011-03-25 | 2012-01-04 | 国云科技股份有限公司 | Cloud storage system and implementation method thereof |
| CN102882981A (en) * | 2012-10-19 | 2013-01-16 | 南京邮电大学 | Load balancing method of node task under equivalence structural cloud platform |
-
2013
- 2013-12-30 CN CN201310745094.0A patent/CN104754001A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101552799A (en) * | 2008-04-04 | 2009-10-07 | 华为技术有限公司 | Media node fault-tolerance method and device |
| US20110099146A1 (en) * | 2009-10-26 | 2011-04-28 | Mcalister Grant Alexander Macdonald | Monitoring of replicated data instances |
| CN101964820A (en) * | 2010-10-08 | 2011-02-02 | 中兴通讯股份有限公司 | Method and system for keeping data consistency |
| CN102307221A (en) * | 2011-03-25 | 2012-01-04 | 国云科技股份有限公司 | Cloud storage system and implementation method thereof |
| CN102882981A (en) * | 2012-10-19 | 2013-01-16 | 南京邮电大学 | Load balancing method of node task under equivalence structural cloud platform |
Cited By (46)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105159919A (en) * | 2015-07-24 | 2015-12-16 | 福建师范大学 | Data multi-copy correlation method and system |
| CN105610864A (en) * | 2016-02-04 | 2016-05-25 | 上海盛巨信息技术有限公司 | Cloud bridge system for connecting NAS storage to cloud storage |
| US10289502B2 (en) | 2016-03-07 | 2019-05-14 | International Business Machines Corporation | User-defined failure domains for software-defined storage systems |
| CN105847364A (en) * | 2016-03-28 | 2016-08-10 | 乐视控股（北京）有限公司 | Public cloud object storage method based on uniform domain name and public cloud object storage system based on uniform domain name |
| CN106202367A (en) * | 2016-07-07 | 2016-12-07 | 腾讯科技（深圳）有限公司 | A kind of processing method and processing device of object information |
| US11281632B2 (en) | 2016-07-07 | 2022-03-22 | Tencent Technology (Shenzhen) Company Limited | Object information processing method and apparatus, and storage medium |
| CN106202396A (en) * | 2016-07-08 | 2016-12-07 | 乐视控股（北京）有限公司 | Object storage method and object storage system |
| CN107528719A (en) * | 2017-03-08 | 2017-12-29 | 深圳市泽云科技有限公司 | A kind of implementation method for lifting cloud storage system high availability |
| CN107018185A (en) * | 2017-03-28 | 2017-08-04 | 清华大学 | The synchronous method and device of cloud storage system |
| CN107154966A (en) * | 2017-04-24 | 2017-09-12 | 浙江宇视科技有限公司 | A kind of video data wiring method and device |
| CN107634853A (en) * | 2017-08-28 | 2018-01-26 | 深圳市云舒网络技术有限公司 | A kind of customizable distributed caching method based on object storage |
| CN107911461A (en) * | 2017-11-24 | 2018-04-13 | 网宿科技股份有限公司 | Object processing method, storage server and cloud storage system in cloud storage system |
| CN109936594A (en) * | 2017-12-15 | 2019-06-25 | 上海欣能信息科技发展有限公司 | A kind of cloud storage system based on electric power data |
| CN109933421A (en) * | 2017-12-18 | 2019-06-25 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus of distributed data processing |
| CN108536795A (en) * | 2018-04-02 | 2018-09-14 | 深圳市华傲数据技术有限公司 | A kind of method, medium and equipment improving diagram data interactive efficiency |
| CN110557975A (en) * | 2018-04-02 | 2019-12-10 | 甲骨文国际公司 | Tenant data comparison for multi-tenant identity cloud services |
| CN108833535A (en) * | 2018-06-14 | 2018-11-16 | 浙江远算云计算有限公司 | User data storage method based on the storage of cloud platform distributed block |
| CN109325025A (en) * | 2018-07-31 | 2019-02-12 | 佛山科学技术学院 | A kind of restructural integrated approach of multi-source heterogeneous platform |
| CN109362236A (en) * | 2018-08-31 | 2019-02-19 | 深圳大学 | Secure storage method of data, device, equipment and storage medium based on double clouds |
| CN109471870A (en) * | 2018-11-16 | 2019-03-15 | 北京金山云网络技术有限公司 | Method, apparatus, electronic equipment and the computer-readable medium that resource data is read |
| CN109471870B (en) * | 2018-11-16 | 2021-07-20 | 北京金山云网络技术有限公司 | Method and device for reading resource data, electronic equipment and computer readable medium |
| US10698618B1 (en) | 2019-03-26 | 2020-06-30 | International Buisness Machines Corporation | User-defined failure domains for software-defined storage systems |
| CN112035402A (en) * | 2019-06-04 | 2020-12-04 | 顺丰科技有限公司 | File storage method and device and terminal equipment |
| CN110287169A (en) * | 2019-06-28 | 2019-09-27 | 四川长虹电器股份有限公司 | A kind of master data moving method between cloud application |
| CN110287169B (en) * | 2019-06-28 | 2022-02-01 | 四川长虹电器股份有限公司 | Main data migration method between cloud applications |
| CN110457325A (en) * | 2019-08-12 | 2019-11-15 | 北京百度网讯科技有限公司 | Method and apparatus for output information |
| CN110457325B (en) * | 2019-08-12 | 2022-05-03 | 北京百度网讯科技有限公司 | Method and apparatus for outputting information |
| CN110457263B (en) * | 2019-08-13 | 2021-10-26 | 北京首都在线科技股份有限公司 | Data storage method and device |
| CN110457263A (en) * | 2019-08-13 | 2019-11-15 | 北京首都在线科技股份有限公司 | A kind of date storage method and device |
| US11675640B2 (en) | 2019-10-29 | 2023-06-13 | Snowflake Inc. | External function invocation by a data system |
| US11876802B2 (en) | 2019-11-14 | 2024-01-16 | Snowflake Inc. | Loading and unloading data at an external storage location |
| CN113228593A (en) * | 2019-11-14 | 2021-08-06 | 斯诺弗雷克公司 | Database integrated external certificateless scratch pad |
| US11522860B2 (en) | 2019-11-14 | 2022-12-06 | Snowflake Inc. | Storage integration with an external storage location |
| CN111107109A (en) * | 2020-01-08 | 2020-05-05 | 世纪恒通科技股份有限公司 | Log-in-free technology based on token |
| CN111309260B (en) * | 2020-02-16 | 2021-04-09 | 西安奥卡云数据科技有限公司 | Data storage node selection method |
| CN111309260A (en) * | 2020-02-16 | 2020-06-19 | 西安奥卡云数据科技有限公司 | Data storage node selection method |
| WO2021189308A1 (en) * | 2020-03-25 | 2021-09-30 | Beijing Didi Infinity Technology And Development Co., Ltd. | Delete operation in object storage system using enhanced meta structure |
| CN111475117A (en) * | 2020-04-09 | 2020-07-31 | 南京信息职业技术学院 | Aggregation cloud storage QoS control method and system |
| CN111475117B (en) * | 2020-04-09 | 2024-01-23 | 南京信息职业技术学院 | Aggregated cloud storage QoS control method and system |
| CN111637891A (en) * | 2020-05-06 | 2020-09-08 | 北京小趣智品科技有限公司 | Toilet positioning method and system |
| WO2021151293A1 (en) * | 2020-08-27 | 2021-08-05 | 平安科技（深圳）有限公司 | Hash ring load balancing method and apparatus, and electronic device and storage medium |
| US11675784B2 (en) | 2021-04-30 | 2023-06-13 | Snowflake Inc. | Configuring parallelism parameters for invocation of external table functions |
| CN114385086A (en) * | 2022-01-17 | 2022-04-22 | 杭州瑞网广通信息技术有限公司 | Data storage method and device based on cloud storage system |
| CN114785795A (en) * | 2022-04-02 | 2022-07-22 | 北京蓝海在线科技有限公司 | Distributed storage method, system, device, storage medium and information processing terminal |
| CN115408396A (en) * | 2022-09-02 | 2022-11-29 | 金蝶征信有限公司 | Business data storage method and device, computer equipment and storage medium |
| CN115408396B (en) * | 2022-09-02 | 2024-04-05 | 金蝶征信有限公司 | Method, device, computer equipment and storage medium for storing business data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104754001A (en) | Cloud storage system and data storage method | |
| US10956601B2 (en) | Fully managed account level blob data encryption in a distributed storage environment | |
| US10764045B2 (en) | Encrypting object index in a distributed storage environment | |
| US10831720B2 (en) | Cloud storage distributed file system | |
| US10789217B2 (en) | Hierarchical namespace with strong consistency and horizontal scalability | |
| US20240179212A1 (en) | Hosted file sync with stateless sync nodes | |
| US10659225B2 (en) | Encrypting existing live unencrypted data using age-based garbage collection | |
| CA2923068C (en) | Method and system for metadata synchronization | |
| CN111078121B (en) | Data migration method and system for distributed storage system and related components | |
| US20190370362A1 (en) | Multi-protocol cloud storage for big data and analytics | |
| US9558194B1 (en) | Scalable object store | |
| KR101434128B1 (en) | Distributed replica storage system with web services interface | |
| US11297031B2 (en) | Hierarchical namespace service with distributed name resolution caching and synchronization | |
| US20190370365A1 (en) | Distributed transactions in cloud storage with hierarchical namespace | |
| CN102420854A (en) | Distributed file system facing to cloud storage | |
| Dwivedi et al. | Analytical review on Hadoop Distributed file system | |
| US11818012B2 (en) | Online restore to different topologies with custom data distribution | |
| CN109597903A (en) | Image file processing apparatus and method, document storage system and storage medium | |
| US10387384B1 (en) | Method and system for semantic metadata compression in a two-tier storage system using copy-on-write | |
| Avilés-González et al. | Scalable metadata management through OSD+ devices | |
| US10628391B1 (en) | Method and system for reducing metadata overhead in a two-tier storage architecture | |
| Branco et al. | Managing very-large distributed datasets | |
| US20240134828A1 (en) | Techniques for efficient encryption and decryption during file system cross-region replication | |
| US20240104062A1 (en) | Techniques for resolving snapshot key inter-dependency during file system cross-region replication |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20150701 |