CN111737052A - Distributed object storage system and method - Google Patents
Distributed object storage system and method Download PDFInfo
- Publication number
- CN111737052A CN111737052A CN202010563948.3A CN202010563948A CN111737052A CN 111737052 A CN111737052 A CN 111737052A CN 202010563948 A CN202010563948 A CN 202010563948A CN 111737052 A CN111737052 A CN 111737052A
- Authority
- CN
- China
- Prior art keywords
- park
- distributed
- master
- index information
- data service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1458—Management of the backup or restore process
- G06F11/1464—Management of the backup or restore process for networked environments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1456—Hardware arrangements for backup
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a distributed object storage system and a method, wherein the system comprises: the system comprises a main park system, at least one spare park system, a message queue and a data copying module; the master park data service module is used for packaging a plurality of objects to be stored into files, storing the files into a distributed file system of a master park system, generating index information of the objects, and adding the index information into a distributed database and a message queue of the master park system; the data replication module is used for acquiring the index information from the message queue, adding the index information to the distributed database of the backup park system, searching the file stored in the distributed file system of the main park system according to the index information, and adding the file to the distributed file system of the backup park system. The invention solves the technical problem that the existing distributed object storage system is not suitable for storing a large number of small objects.
Description
Technical Field
The invention relates to the technical field of object storage, in particular to a distributed object storage system and a distributed object storage method.
Background
Object storage is a technology often used in the internet, and is different from file storage, and objects are not organized in a directory hierarchy structure. Each object is located at the same level of a flat space called a storage pool, and each element at each level has a unique identification in the storage system by which a user accesses a container or object. In object storage, nested folders are generally discarded using a flat data organization structure, thereby avoiding the maintenance of a large directory tree.
At present, the object storage usually adopts a distributed storage mode. However, distributed object storage has inherent deficiencies. If the object is stored in a distributed file system (HDFS) as a separate file, the number of stored files has an upper limit due to the memory limitation of a named node (NameNode), and a single object is usually small, and a large number of small files are stored, which may greatly waste the storage space of the entire distributed file system. In addition, most of the existing file systems have poor disaster tolerance capability, and once accidents such as power failure occur, data cannot be read at all, and the phenomenon of object loss is easy to occur.
Disclosure of Invention
In order to solve the technical problems in the background art, the invention provides a distributed object storage system and a method.
To achieve the above object, according to one aspect of the present invention, there is provided a distributed object storage system including: main garden system, at least one spare park system, message queue and data replication module, wherein:
the master park system and the backup park system respectively comprise a distributed file system and a distributed database, and the master park system also comprises at least one master park data service module;
the master park data service module is used for packaging a plurality of objects to be stored into files, storing the files into a distributed file system of the master park system, generating index information of the objects, and adding the index information into a distributed database of the master park system and the message queue;
the data replication module is used for acquiring the index information from the message queue, adding the index information to the distributed database of the backup park system, searching the file stored in the distributed file system of the main park system according to the index information, and adding the file to the distributed file system of the backup park system.
Optionally, the park preparing system further includes at least one park preparing data service module;
the master park data service module and the backup park data service module are used for receiving an object query request of a user, querying index information corresponding to the object query request from a distributed database of a park system, querying an object corresponding to the object query request from a file stored in a distributed file system of the park system according to the queried index information, and sending the queried object to the user.
Optionally, the distributed object storage system further includes:
and the coordination module is used for receiving an object query request sent by a user and distributing the object query request sent by the user to a random one of the main park data service module and the standby park data service module which are normal in current working state for processing.
Optionally, the distributed object storage system further includes:
and the coordination module is used for receiving an object storage request sent by a user and distributing the object storage request to a random one of the master park data service modules with a normal current working state for processing, wherein the object storage request comprises at least one object to be stored.
Optionally, the coordination module is further configured to receive an object deletion request or an object modification request sent by a user, and allocate the object deletion request or the object modification request to a random one of the master campus data service modules whose current working state is normal for processing.
Optionally, the master park data service module is further configured to delete, when the object deletion request is received, the index information corresponding to the object deletion request stored in the distributed database of the master park system.
Optionally, the master park data service module is further configured to, when receiving the object modification request, store the modified object corresponding to the modified target object in the distributed file system of the master park system, generate index information of the modified object, and update the index information of the modified target object stored in the distributed database of the master park system to the index information of the modified object.
In order to achieve the above object, according to another aspect of the present invention, there is provided a distributed object storage method, including:
a master park data service module of a master park system packs a plurality of objects to be stored into files and stores the files into a distributed file system of the master park system;
the master park data service module generates index information of the object, and adds the index information into a distributed database of the master park system and a message queue;
the data replication module acquires the index information from the message queue and adds the index information to a distributed database of the backup park system;
and the data replication module searches the files stored in the distributed file system of the master park system according to the index information and adds the files to the distributed file system of the backup park system.
The invention has the beneficial effects that: the distributed object storage system of the invention stores the objects in multiple parks, can support high availability and high concurrency, and has very high disaster tolerance capability. According to the invention, the objects which are not suitable for being directly uploaded to the distributed system are combined into the HDFS file for storage, so that the distributed object storage system can store a large number of objects, and the flexibility and the expandability of the distributed object storage system are effectively improved. By the scheme, the technical problem that the existing distributed object storage system is not suitable for storing a large number of small objects is solved, and the technical effects of flexibility and expansibility of the system are effectively improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts. In the drawings:
FIG. 1 is a schematic diagram of a distributed object storage system according to an embodiment of the present invention;
FIG. 2 is a flow chart of a distributed object storage method according to an embodiment of the present invention;
FIG. 3 is a flow diagram of querying an object in an embodiment of the invention;
FIG. 4 is a flow chart of an embodiment of the present invention for modifying an object.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
It should be noted that the terms "comprises" and "comprising," and any variations thereof, in the description and claims of the present invention and the above-described drawings, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
Fig. 1 is a schematic diagram of a distributed object storage system according to an embodiment of the present invention, and as shown in fig. 1, the distributed object storage system according to the embodiment includes: the system comprises a main park system, at least one spare park system, a message queue and a data replication module (Replicator).
As shown in fig. 1, each of the master park system and the backup park system includes: a distributed file system (HDFS) and a distributed database (in fig. 1, the Hbase database is taken as an example), in addition, the master park system further includes at least one master park data service module (master park Worker), and the backup park system further includes at least one backup park data service module (backup park Worker).
In an optional embodiment of the present invention, the master park system may process an addition, deletion, search, and modification request of a user for an object, and the backup park system may process only an object query request of the user.
When an object storage request of a user is received, a master park data service module packs a plurality of objects to be stored into a file (HDFS file) and stores the file into a distributed file system (HDFS) of a master park system, wherein the object storage request comprises at least one object to be stored. And simultaneously, a master park data service module (master park Worker) generates index information of the object, and adds the index information into a distributed database of a master park system and a message queue. In the embodiment of the present invention, the index information corresponding to each object includes file information stored by the object and an offset of the object in the corresponding file, and it is subsequently possible to determine in which file in the distributed file system the object is stored and a location in the file according to the index information.
The data replication module (Replicator) is used for realizing data synchronization of the backup park system and the main park system, namely ensuring that files and objects stored in a distributed file system (HDFS) of the backup park system and index information stored in a distributed database of the backup park system are the same as those of the main park system.
Specifically, the data replication module (Replicator) acquires the index information of the object from the message queue, adds the index information to the distributed database of the backup park system, searches a file corresponding to the index information from the distributed file system of the master park system, and adds the file to the distributed file system of the backup park system. Because a plurality of objects correspond to one file, when the file is added to the distributed file system of the backup park system, whether the file exists in the distributed file system of the backup park system needs to be judged, and if the file exists, the addition is abandoned.
In an alternative embodiment of the present invention, the message queue may be a Kafka message queue, and the distributed database may be an HBase database.
The invention considers that if the object is stored in the distributed file system HDFS as a single file, the hardware resource is greatly wasted. If a plurality of objects are combined into a large file to be stored in the HDFS, the problem of overlarge consumption of hardware resources is solved, but searching for each object needs to search for the large file once, and the time consumed for reading the file is difficult to meet the requirements of users. Therefore, the invention combines a plurality of objects into a file with fixed size and stores the file in the HDFS, and stores index information such as metadata of the objects, offsets of the objects in the file and the like in the distributed database HBase. The HBase supports real-time super-large scale random access, so that when a user reads an object, the system reads the index information in the HBase firstly, then reads the object at a corresponding position according to the index information and returns the object to the user. By doing so, not only can the storage space be saved, but also the time consumption for reading the object can be reduced.
As shown in fig. 1, the distributed object storage system of the embodiment of the present invention further includes a Coordinator module (Coordinator). The coordination module (Coordinator) is configured to receive an object operation request sent by a user through a Client (Client), where the object operation request includes: object storage requests, object query requests, object deletion requests, and object modification requests. The master park system can process all types of object operation requests, and the standby park system can only process object query requests of users.
In the embodiment of the present invention, after the master campus data service module (master campus Worker) and the backup campus data service module (backup campus Worker) are started, heartbeat information is periodically sent to the coordination module (Coordinator), which indicates that they are in a normal working state. After receiving the heartbeat information, a coordination module (Coordinator) stores the heartbeat information in a Redis memory database, and periodically deletes expired heartbeat information, so that a master park Worker and a standby park Worker which are in normal working states are ensured to be stored in the Redis.
The coordination module (Coordinator) is used for allocating an object query request sent by a user to one of the master park Worker and the standby park Worker for processing, and during allocation, the coordination module (Coordinator) reads the working states of each master park Worker and each standby park Worker from Redis and randomly selects one of each master park Worker and each standby park Worker with normal working states to allocate the object query request.
The coordination module (Coordinator) is further configured to allocate an object storage request, an object deletion request, and an object modification request sent by a user to one of the master campus Workers for processing, and during allocation, the coordination module (Coordinator) reads a working state of each master campus Worker from the Redis cache, and randomly selects a request for allocating the user from each master campus Worker with a normal working state.
In an optional embodiment, the coordination module (Coordinator) further obtains busy and idle states of each master park Worker and each backup park Worker in real time. When the object query request of the user is distributed, the object query request is distributed to a random one of each master park Worker and each standby park Worker with a normal working state and an idle busy state for processing. When an object storage request, an object deletion request and an object modification request of a user are allocated, the requests are allocated to a random one of the master park workers with normal working state and idle busy/idle state for processing.
When a main park data service module (a main park Worker) processes an object query request of a user, index information corresponding to a query target object is queried from a distributed database (Hbase) of a main park system, wherein the object query request comprises query target object information. And then the main park data service module (main park Worker) inquires the inquiry target object from the files stored in the distributed file system (HDFS) of the main park system according to the inquired index information, and sends the inquired inquiry target object to the user.
When a backup park data service module (backup park Worker) processes an object query request of a user, index information corresponding to a query target object is queried from a distributed database (Hbase) of a backup park system. And then the data service module of the spare park (spare park Worker) inquires the inquiry target object from the file stored in the distributed file system (HDFS) of the spare park system according to the inquired index information, and sends the inquired inquiry target object to the user.
When a master park data service module (master park Worker) processes an object deletion request of a user, deleting index information corresponding to an object to be deleted stored in a distributed database (Hbase) of a master park system, wherein the object deletion request comprises information of the object to be deleted.
When a master park data service module (master park Worker) processes an object modification request of a user, a modified object corresponding to a modified target object is stored in a distributed file system (HDFS) of a master park system. And meanwhile, generating index information of the modified object, and updating the index information of the modified target object stored in a distributed database (Hbase) of the main park system into the index information of the modified object.
It can be seen from the above embodiments that the present invention provides a distributed object storage system, which stores objects in multiple parks, supports high availability and high concurrency, and has high disaster tolerance capability. The system receives the object storage request, and writes the object in the object storage request into the object storage system for object storage. In the mode, a plurality of objects which are not suitable for being directly uploaded to the distributed object storage system for object storage are combined into the HDFS file, and the objects are stored in the distributed object storage system, so that the distributed object storage system can store a large number of objects, and the flexibility and the expandability of the distributed object storage system are effectively improved. By the scheme, the technical problem that the existing distributed object storage system is not suitable for storing a large number of small objects is solved, and the technical effects of flexibility and expansibility of the system are effectively improved.
It should be noted that, as used above, the term "unit" or "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the above embodiments are preferably implemented in software, an implementation in hardware or a combination of software and hardware is also possible and contemplated.
Based on the same inventive concept, the embodiment of the present invention further provides a distributed object storage method, as described in the following embodiments. Because the principle of the distributed object storage method for solving the problem is similar to that of the distributed object storage system, the embodiment of the distributed object storage method can be referred to as the embodiment of the distributed object storage system, and repeated details are not repeated.
Fig. 2 is a flowchart of a distributed object storage method according to an embodiment of the present invention, and as shown in fig. 2, the distributed object storage method according to the embodiment of the present invention includes step S101 to step S104.
Step S101, a master park data service module of a master park system packs a plurality of objects to be stored into files and stores the files into a distributed file system of the master park system.
And step S102, the data service module of the main park generates index information of the object, and the index information is added into a distributed database and a message queue of the main park system.
And step S103, the data replication module acquires the index information from the message queue and adds the index information to a distributed database of the park backup system.
And step S104, the data replication module searches the files stored in the distributed file system of the master park system according to the index information and adds the files to the distributed file system of the backup park system.
Fig. 3 is a flowchart of querying an object according to an embodiment of the present invention, and as shown in fig. 3, the flowchart of querying an object according to the present invention includes step S201 and step S202.
Step S201, the main park data service module and the park data service module of the park system inquire index information corresponding to the object inquiry request from a distributed database of the park system according to the object inquiry request of the user.
Step S202, the main park data service module and the backup park data service module inquire the object corresponding to the object inquiry request from the file stored in the distributed file system of the park system according to the inquired index information, and send the inquired object to the user.
In an optional embodiment of the present invention, the distributed object storage method of the present invention further includes:
and the coordination module receives an object query request sent by a user, and distributes the object query request sent by the user to a random one of the master park data service module and the backup park data service module which are in normal working states for processing.
In an optional embodiment of the present invention, the distributed object storage method of the present invention further includes:
the coordination module receives an object storage request sent by a user, and distributes the object storage request to a random one of the master park data service modules with normal working state for processing, wherein the object storage request comprises at least one object to be stored.
In an optional embodiment of the present invention, the distributed object storage method of the present invention further includes:
the coordination module receives an object deletion request or an object modification request sent by a user, and distributes the object deletion request or the object modification request to a random one of the main park data service modules with normal current working state for processing.
In an optional embodiment of the present invention, the distributed object storage method of the present invention further includes:
and when receiving the object deletion request, the master park data service module deletes the index information corresponding to the object deletion request stored in the distributed database of the master park system.
Fig. 4 is a flowchart of modifying an object according to an embodiment of the present invention, and as shown in fig. 4, in an alternative embodiment of the present invention, the process of modifying an object includes step S301 and step S302.
Step S301, when receiving the object modification request, the master park data service module stores the modified object corresponding to the modified target object in the distributed file system of the master park system.
Step S302, the master park data service module generates index information of the modified object, and updates the index information of the modified target object stored in the distributed database of the master park system to the index information of the modified object.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (14)
1. A distributed object storage system, comprising: main garden system, at least one spare park system, message queue and data replication module, wherein:
the master park system and the backup park system respectively comprise a distributed file system and a distributed database, and the master park system also comprises at least one master park data service module;
the master park data service module is used for packaging a plurality of objects to be stored into files, storing the files into a distributed file system of the master park system, generating index information of the objects, and adding the index information into a distributed database of the master park system and the message queue;
the data replication module is used for acquiring the index information from the message queue, adding the index information to the distributed database of the backup park system, searching the file stored in the distributed file system of the main park system according to the index information, and adding the file to the distributed file system of the backup park system.
2. The distributed object storage system of claim 1, wherein the park system further comprises at least one park data service module;
the master park data service module and the backup park data service module are used for receiving an object query request of a user, querying index information corresponding to the object query request from a distributed database of a park system, querying an object corresponding to the object query request from a file stored in a distributed file system of the park system according to the queried index information, and sending the queried object to the user.
3. The distributed object storage system of claim 2, further comprising:
and the coordination module is used for receiving an object query request sent by a user and distributing the object query request sent by the user to a random one of the main park data service module and the standby park data service module which are normal in current working state for processing.
4. The distributed object storage system of claim 1, further comprising:
and the coordination module is used for receiving an object storage request sent by a user and distributing the object storage request to a random one of the master park data service modules with a normal current working state for processing, wherein the object storage request comprises at least one object to be stored.
5. The distributed object storage system of claim 4,
the coordination module is further configured to receive an object deletion request or an object modification request sent by a user, and allocate the object deletion request or the object modification request to a random one of the master park data service modules whose current working state is normal for processing.
6. The distributed object storage system of claim 5,
the master park data service module is further configured to delete the index information corresponding to the object deletion request stored in the distributed database of the master park system when the object deletion request is received.
7. The distributed object storage system of claim 5,
the master park data service module is further configured to, when receiving the object modification request, store the modified object corresponding to the modified target object in the distributed file system of the master park system, generate index information of the modified object, and update the index information of the modified target object stored in the distributed database of the master park system to the index information of the modified object.
8. A distributed object storage method, comprising:
a master park data service module of a master park system packs a plurality of objects to be stored into files and stores the files into a distributed file system of the master park system;
the master park data service module generates index information of the object, and adds the index information into a distributed database of the master park system and a message queue;
the data replication module acquires the index information from the message queue and adds the index information to a distributed database of the backup park system;
and the data replication module searches the files stored in the distributed file system of the master park system according to the index information and adds the files to the distributed file system of the backup park system.
9. The distributed object storage method of claim 8, further comprising:
the master park data service module and the park data service module of the park system inquire index information corresponding to the object inquiry request from a distributed database of the park system according to the object inquiry request of a user;
and the main park data service module and the standby park data service module inquire the object corresponding to the object inquiry request from the file stored in the distributed file system of the park system according to the inquired index information, and send the inquired object to the user.
10. The distributed object storage method of claim 9, further comprising:
and the coordination module receives an object query request sent by a user, and distributes the object query request sent by the user to a random one of the master park data service module and the backup park data service module which are in normal working states for processing.
11. The distributed object storage method of claim 8, further comprising:
the coordination module receives an object storage request sent by a user, and distributes the object storage request to a random one of the master park data service modules with normal working state for processing, wherein the object storage request comprises at least one object to be stored.
12. The distributed object storage method of claim 11, further comprising:
the coordination module receives an object deletion request or an object modification request sent by a user, and distributes the object deletion request or the object modification request to a random one of the main park data service modules with normal current working state for processing.
13. The distributed object storage method of claim 12, further comprising:
and when receiving the object deletion request, the master park data service module deletes the index information corresponding to the object deletion request stored in the distributed database of the master park system.
14. The distributed object storage method of claim 12, further comprising:
when the master park data service module receives the object modification request, storing a modified object corresponding to a modified target object into a distributed file system of the master park system;
the master park data service module generates index information of the modified object and updates the index information of the modified target object stored in the distributed database of the master park system to the index information of the modified object.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010563948.3A CN111737052B (en) | 2020-06-19 | 2020-06-19 | Distributed object storage system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010563948.3A CN111737052B (en) | 2020-06-19 | 2020-06-19 | Distributed object storage system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111737052A true CN111737052A (en) | 2020-10-02 |
| CN111737052B CN111737052B (en) | 2023-07-07 |
Family
ID=72650154
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010563948.3A Active CN111737052B (en) | 2020-06-19 | 2020-06-19 | Distributed object storage system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111737052B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112764973A (en) * | 2021-01-28 | 2021-05-07 | 重庆紫光华山智安科技有限公司 | Data disaster tolerance method and device, electronic equipment and readable storage medium |
| CN113127567A (en) * | 2021-04-29 | 2021-07-16 | 武汉优品楚鼎科技有限公司 | Financial data computing system, method and device based on distributed storage |
| CN116340273A (en) * | 2023-03-27 | 2023-06-27 | 中国人民财产保险股份有限公司 | Heterogeneous file synchronous extraction method, device and equipment based on different storage modes |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104731516A (en) * | 2013-12-18 | 2015-06-24 | 腾讯科技（深圳）有限公司 | Method and device for accessing files and distributed storage system |
| CN105320773A (en) * | 2015-11-03 | 2016-02-10 | 中国人民解放军理工大学 | Distributed duplicated data deleting system and method based on Hadoop platform |
| CN107861686A (en) * | 2017-09-26 | 2018-03-30 | 深圳前海微众银行股份有限公司 | File memory method, service end and computer-readable recording medium |
| CN108460072A (en) * | 2017-12-26 | 2018-08-28 | 北京国电通网络技术有限公司 | With electricity consumption data retrieval method and system |
-
2020
- 2020-06-19 CN CN202010563948.3A patent/CN111737052B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104731516A (en) * | 2013-12-18 | 2015-06-24 | 腾讯科技（深圳）有限公司 | Method and device for accessing files and distributed storage system |
| CN105320773A (en) * | 2015-11-03 | 2016-02-10 | 中国人民解放军理工大学 | Distributed duplicated data deleting system and method based on Hadoop platform |
| CN107861686A (en) * | 2017-09-26 | 2018-03-30 | 深圳前海微众银行股份有限公司 | File memory method, service end and computer-readable recording medium |
| CN108460072A (en) * | 2017-12-26 | 2018-08-28 | 北京国电通网络技术有限公司 | With electricity consumption data retrieval method and system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112764973A (en) * | 2021-01-28 | 2021-05-07 | 重庆紫光华山智安科技有限公司 | Data disaster tolerance method and device, electronic equipment and readable storage medium |
| CN113127567A (en) * | 2021-04-29 | 2021-07-16 | 武汉优品楚鼎科技有限公司 | Financial data computing system, method and device based on distributed storage |
| CN116340273A (en) * | 2023-03-27 | 2023-06-27 | 中国人民财产保险股份有限公司 | Heterogeneous file synchronous extraction method, device and equipment based on different storage modes |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111737052B (en) | 2023-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111737052B (en) | Distributed object storage system and method | |
| CN110213352B (en) | Method for aggregating dispersed autonomous storage resources with uniform name space | |
| CN101354726B (en) | Method for managing memory metadata of cluster file system | |
| CN106294190B (en) | Storage space management method and device | |
| CN101799797B (en) | Dynamic allocation method of user disk quota in distributed storage system | |
| CN102013991B (en) | Method, management equipment and system for automatically expanding capacity | |
| US20040205109A1 (en) | Computer system | |
| CN102694860A (en) | Method, equipment and system for data processing of cloud storage | |
| CN103067461A (en) | Metadata management system of document and metadata management method thereof | |
| CN103647797A (en) | Distributed file system and data access method thereof | |
| CN112162846B (en) | Transaction processing method, device and computer readable storage medium | |
| CN102236580B (en) | Method for distributing node to ETL (Extraction-Transformation-Loading) task and dispatching system | |
| CN102333029A (en) | Routing method in server cluster system | |
| CN109947787A (en) | A kind of storage of data hierarchy, hierarchical query method and device | |
| CN106326239A (en) | Distributed file system and file meta-information management method thereof | |
| CN101472166A (en) | Method for caching and enquiring content as well as point-to-point medium transmission system | |
| CN105159845A (en) | Memory reading method | |
| CN110022338B (en) | File reading method and system, metadata server and user equipment | |
| CN105205143A (en) | File storage and processing method, device and system | |
| CN103365740B (en) | A kind of data cold standby method and device | |
| CN108319634B (en) | Directory access method and device for distributed file system | |
| CN114594914A (en) | Control method and system for distributed storage system | |
| CN109871365A (en) | A kind of distributed file system | |
| KR101341412B1 (en) | Apparatus and method of controlling metadata in asymmetric distributed file system | |
| CN107493309B (en) | File writing method and device in distributed system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |