CN112100129A - Data access method, data storage method, data access device and file storage system - Google Patents
Data access method, data storage method, data access device and file storage system Download PDFInfo
- Publication number
- CN112100129A CN112100129A CN202010965297.0A CN202010965297A CN112100129A CN 112100129 A CN112100129 A CN 112100129A CN 202010965297 A CN202010965297 A CN 202010965297A CN 112100129 A CN112100129 A CN 112100129A
- Authority
- CN
- China
- Prior art keywords
- file
- data
- storage
- directory management
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013500 data storage Methods 0.000 title claims abstract description 105
- 238000000034 method Methods 0.000 title claims abstract description 75
- 238000007726 management method Methods 0.000 claims abstract description 382
- 239000012634 fragment Substances 0.000 claims abstract description 164
- 230000002688 persistence Effects 0.000 claims description 24
- 230000001360 synchronised effect Effects 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 4
- 238000013507 mapping Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data access method, a data storage device and a file storage system; the data access method comprises the following steps: determining a file directory management fragment to which a first target data volume to which a file to be accessed belongs based on a corresponding relation between the file directory management fragment and the data volume in a file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment manages metadata of files in the designated data volume; and accessing the file to be accessed based on the metadata of the file to be accessed, which is acquired from the file directory management fragment to which the first target data volume belongs. In the mode, the file storage system is provided with the plurality of file directory management fragments, and when the file volume is large, the file target management fragments can be expanded horizontally, so that the problem that the number of metadata of the stored files in a single file directory management node is limited is solved, and the number of the stored files of the system and the total data volume are increased.
Description
Technical Field
The present invention relates to the field of file storage technologies, and in particular, to a data access method, a data storage device, and a file storage system.
Background
HDFS (Hadoop Distributed File System) is a typical open source Distributed File storage System which is widely used at present. In the HDFS system, metadata of all files are stored in a NameNode node, and the memory space of the NameNode node is limited, so that the quantity of the metadata of the files which can be stored in the NameNode node is limited, and the total quantity of the files and the total data quantity which can be stored in the whole system are limited.
Disclosure of Invention
In view of the above, the present invention provides a data access method, a data storage device, and a file storage system, so as to increase the number of files stored in the system and the total data volume.
In a first aspect, an embodiment of the present invention provides a data access method, where the method includes: determining a first target data volume to which a file to be accessed belongs based on the data access request; determining a file directory management fragment to which the first target data volume belongs based on a corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume; acquiring metadata of a file to be accessed from a file directory management fragment to which a first target data volume belongs; and accessing the file to be accessed based on the metadata of the file to be accessed.
Furthermore, each file directory management fragment comprises a plurality of file directory management nodes; the file directory management nodes comprise a main file directory management node and at least one standby file directory management node; the main file directory management node is used for providing metadata of a file to be accessed; and the standby file directory management node synchronizes the data of the main file directory management node.
Further, a main file directory management node in the file directory management nodes is determined based on a preset raft protocol; and the data of the standby file directory management node is synchronized with the data of the main file directory management node through a preset raft protocol.
Furthermore, a plurality of file storage clusters are arranged in the file storage system; the metadata of the file to be accessed comprises a file storage cluster where the file data of the file to be accessed is located; the step of accessing the file to be accessed based on the metadata of the file to be accessed comprises: and acquiring the file to be accessed from the file storage cluster to which the file to be accessed belongs.
Further, the file storage cluster comprises a cluster management node and a data storage node; the cluster management node stores the corresponding relation between the file data of each file stored in the file storage cluster and the data storage node; the method for acquiring the file to be accessed from the file storage cluster to which the file to be accessed belongs comprises the following steps: determining a target data storage node where file data of a file to be accessed is located through a cluster management node of a file storage cluster to which the file to be accessed belongs; and acquiring file data of the file to be accessed from the target data storage node.
Further, each file storage cluster comprises a plurality of cluster management nodes; the plurality of cluster management nodes comprise a main cluster management node and at least one standby cluster management node; the main cluster management node is used for providing a target data storage node where file data of a file to be accessed is located; and the data of the standby cluster management node and the data of the main cluster management node are synchronized.
Further, a data persistence service is set in the file storage system; the data persistence service is used for providing a corresponding relation between the file directory management fragments and the data volumes; before the step of determining, based on the data access request, a first target data volume to which the file to be accessed belongs, the method further includes: and acquiring the corresponding relation between the file directory management fragments and the data volumes through data persistence service.
In a second aspect, an embodiment of the present invention provides a data storage method, where the method includes: determining a second target data volume to which the file to be stored belongs based on the data storage request; determining a file directory management fragment to which the second target data volume belongs based on a corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume; determining a target storage position of a file to be stored; generating metadata of a file to be stored in a file directory management fragment to which a second target data volume belongs based on the target storage position; and storing the file to be stored to a target storage position.
Furthermore, a plurality of file storage clusters are arranged in the file storage system; the target storage location includes: a target file storage cluster storing file data of a file to be stored; the method for storing the file to be stored in the target storage position comprises the following steps: and storing the file to be stored to the target file storage cluster.
Further, the file storage cluster comprises a cluster management node and a data storage node; the cluster management node stores the corresponding relation between the file data of each file stored in the file storage cluster and the data storage node; the method for storing the file to be stored in the target file storage cluster comprises the following steps: determining a target data storage node for storing a file to be stored from a target file storage cluster; based on the target data storage node, generating a corresponding relation between file data of a file to be stored and the target data storage node in a cluster management node of a target file storage cluster; and storing the file data of the file to be stored to the target data storage node.
In a third aspect, an embodiment of the present invention provides a file storage system, where the system includes a data request server, multiple file directory management fragments, and multiple file storage clusters; each file storage cluster is used for storing files in a specified data volume; each file directory management fragment is used for managing metadata of files in the designated data volume; the metadata is used for indicating a file storage cluster where the files in the specified data volume are located; and the data request server is used for managing the corresponding relation between the fragments and the data volumes based on the file directory and processing the received data request.
Further, a data persistence service is set in the file storage system; the data persistence service is used for providing the corresponding relation between the file directory management fragments and the data volumes to the data request server side.
Furthermore, each file directory management fragment comprises a plurality of file directory management nodes; the file directory management nodes comprise a main file directory management node and at least one standby file directory management node; the main file directory management node is used for providing metadata of files in the designated data volume; and the standby file directory management node synchronizes the data of the main file directory management node.
Further, the file storage cluster comprises a cluster management node and a data storage node; the data storage nodes are used for storing file data of each file in the file storage cluster; the cluster management node stores the corresponding relation between the file data of each file stored in the file storage cluster and the data storage node.
Further, the file storage cluster comprises a plurality of cluster management nodes; the plurality of cluster management nodes comprise a main cluster management node and at least one standby cluster management node; the main cluster management node is used for providing a data storage node where file data of each file in the file storage cluster is located; and the data of the standby cluster management node and the data of the main cluster management node are synchronized.
In a fourth aspect, an embodiment of the present invention provides a data access apparatus, where the apparatus includes: the first data volume determining module is used for determining a first target data volume to which a file to be accessed belongs based on the data access request; the first management fragment determining module is used for determining the file directory management fragment to which the first target data volume belongs based on the corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume; the file access module is used for acquiring metadata of a file to be accessed from the file directory management fragment to which the first target data volume belongs; and accessing the file to be accessed based on the metadata of the file to be accessed.
In a fifth aspect, an embodiment of the present invention provides a data storage apparatus, where the apparatus includes: the second data volume determining module is used for determining a second target data volume to which the file to be stored belongs based on the data storage request; the second management fragment determining module is used for determining the file directory management fragment to which the second target data volume belongs based on the corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume; the file storage module is used for determining a target storage position of a file to be stored; generating metadata of a file to be stored in a file directory management fragment to which a second target data volume belongs based on the target storage position; and storing the file to be stored to a target storage position.
In a sixth aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, where the memory stores machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to implement the data access method of any one of the first aspect, the data storage method of any one of the second aspect, or the file storage system of any one of the third aspect.
In a seventh aspect, an embodiment of the present invention provides a machine-readable storage medium storing machine-executable instructions, which when invoked and executed by a processor, cause the processor to implement the data access method of any one of the first aspect, the data storage method of any one of the second aspect, or the file storage system of any one of the third aspect.
The embodiment of the invention has the following beneficial effects:
the embodiment of the invention provides a data access method, a data storage device and a file storage system, and the method comprises the steps of determining a first target data volume to which a file to be accessed belongs based on a data access request; determining a file directory management fragment to which the first target data volume belongs based on a corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume; acquiring metadata of a file to be accessed from a file directory management fragment to which a first target data volume belongs; and accessing the file to be accessed based on the metadata of the file to be accessed. In the mode, the file storage system is provided with the plurality of file directory management fragments, and when the file volume is large, the file target management fragments can be expanded horizontally, so that the problem that the number of metadata of the stored files in a single file directory management node is limited is solved, and the number of the stored files of the system and the total data volume are increased.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a data access method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a file storage system according to an embodiment of the present invention;
FIG. 3 is a flow chart of another data access method provided by an embodiment of the invention;
FIG. 4 is a flowchart of a data storage method according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a file storage system according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a data access device according to an embodiment of the present invention;
FIG. 7 is a schematic structural diagram of a data storage device according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
HDFS is currently a more typical, widely used, open source distributed file storage system. HDFS has several major drawbacks: firstly, the size of the stored file is limited: all file metadata are stored in the NameNode node, and the NameNode node has no horizontal capacity expansion capability, so that the number of files and the total data volume which can be stored by a cluster are limited by the bottleneck of a memory of the NameNode node, and generally only hundreds of millions of files can be stored; secondly, the reliability of metadata is not high: the NameNode node of the HDFS comprises a main node and a standby node, the metadata synchronization of the main node and the standby node is asynchronous and synchronous, when the metadata is updated, the main node is down when the main node is synchronously updated to the standby node or not synchronously updated to the slave main node, and part of the latest metadata can be lost; thirdly, service availability is not high: after a main NameNode node of the HDFS is hung, a standby NameNode node can provide services to the outside only after all metadata are loaded into a memory, and when the file scale is hundreds of millions of levels, the switching time is minutes. Based on this, the data access method, the data storage device and the file storage system provided by the embodiments of the present invention may be applied to various hardware devices such as a computer and a notebook, and particularly may be applied to a device having a data access function. The techniques may be implemented by associated hardware and software. It should be noted that the file storage system in this embodiment may specifically be a distributed file storage system.
To facilitate understanding of the embodiment, a detailed description is first made of a data access method disclosed in the embodiment of the present invention, as shown in fig. 1, the method includes the following steps:
step S102, determining a first target data volume to which a file to be accessed belongs based on a data access request;
the data volume can be a name space used for storing directories, files and the like created by a user; before using a file storage system, a user firstly needs to create a data volume of a file to be accessed; in actual implementation, a user may send a data access request through a device such as a client, where the data access request includes a data volume name specified by the user, and after receiving the data access request at a server of the file storage system, a first target data volume to which a file to be accessed belongs may be directly determined through the data volume name included in the data access request. The server of the file storage system usually exposes operation interfaces such as volumes, directories and files to the outside, and forwards a data access request of a user to a corresponding module for processing.
Step S104, determining a file directory management fragment to which the first target data volume belongs based on the corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume;
the file storage system mentioned in this embodiment may be understood as a file system with a directory tree, where a user stores files uploaded by the user and provides typical file system operations to the user, such as: volume operations, directory operations, file links, rights and owners to change files, etc. The file directory management fragment may be referred to as Slice, and a plurality of file directory management fragments, for example, three or 5 file directory management fragments, are provided in the file storage system; each file directory management fragment logically comprises a plurality of data volumes, each data volume corresponds to a directory tree, and each data volume only belongs to one file directory management fragment and cannot cross the file directory management fragments. Each file directory management fragment is used for managing metadata of files in the designated data volume; metadata, which may also be referred to as inode ids, each file has unique metadata, which is also a unique identification of the file, and may be defined as data describing the file, descriptive information about the file and information resources.
Each data volume only belongs to one file directory management fragment, so that the corresponding relation between the file directory management fragments and the data volumes is a one-to-one corresponding relation, and the corresponding relation can be stored through a specified service in a file storage system; specifically, after determining a first target data volume to which a file to be accessed belongs, a server of the file storage system may first find a file directory management fragment obtained from the plurality of data volumes by directly using a correspondence between the file directory management fragment and the data volume, and then determine the file directory management fragment to which the first target data volume belongs.
Step S106, obtaining metadata of a file to be accessed from a file directory management fragment to which the first target data volume belongs; and accessing the file to be accessed based on the metadata of the file to be accessed.
Since each file directory management partition is used to manage metadata for files in a specified data volume; that is, each file directory management fragment includes metadata of a corresponding file; therefore, after determining the file directory management fragment to which the first target data volume belongs, the metadata of the file to be accessed can be obtained through the file directory management fragment, and specifically, the metadata of the file to be accessed can be obtained through a plurality of file directory management nodes included in the file directory management fragment; the file directory management node is mainly used for managing a directory tree of a user and mapping of files to metadata, and can provide metadata of files to be accessed.
The embodiment of the invention provides a data access method, which comprises the steps of determining a first target data volume to which a file to be accessed belongs based on a data access request; determining a file directory management fragment to which the first target data volume belongs based on a corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume; acquiring metadata of a file to be accessed from a file directory management fragment to which a first target data volume belongs; and accessing the file to be accessed based on the metadata of the file to be accessed. In the mode, the file storage system is provided with the plurality of file directory management fragments, and when the file volume is large, the file target management fragments can be expanded horizontally, so that the problem that the number of metadata of the stored files in a single file directory management node is limited is solved, and the number of the stored files of the system and the total data volume are increased.
Each file directory management fragment comprises a plurality of file directory management nodes; the file directory management nodes comprise a main file directory management node and at least one standby file directory management node; the main file directory management node is used for providing metadata of a file to be accessed; and the standby file directory management node synchronizes the data of the main file directory management node.
Specifically, referring to the schematic structural diagram of the file storage system shown in fig. 2, the plurality of file directory management fragments correspond to Slice1, Slice2, etc. in the figure; the file directory management node corresponds to the NameNode in the graph; the main file directory management node corresponds to a Master NameNode in the graph, and the standby file directory management node corresponds to a Slave NameNode in the graph; the data of the plurality of file directory management nodes may be completely mirrored.
The file directory management node provides metadata of the file to be accessed mainly through the mapping relation from the file to the metadata. The data of the standby file directory management node and the data of the main file directory management node can be asynchronous and synchronous.
It should be noted that the file directory management fragments included in the file storage system are not limited to the number shown in the figure, and may also be more, and usually include three or five, and the number is not limited herein, and the specific number may be set according to actual needs. In addition, the number of the at least one standby file directory management node included in the file directory management node is not limited to the number shown in the drawing, and may also be more, and the specific number is not limited herein, and may be set according to actual needs.
In addition, a main file directory management node in the file directory management nodes is determined based on a preset raft protocol; and the data of the standby file directory management node is synchronized with the data of the main file directory management node through a preset raft protocol.
The raft protocol can enable servers of one cluster to form a copy state machine, and maintain the consistency of data of a plurality of file directory management nodes; usually, in an initial state, a plurality of file directory management nodes in each file directory management fragment have no difference between active and standby, one active file directory management node is selected from the plurality of file directory management nodes through an election mechanism of a raft protocol, and the rest file directory management nodes directly become standby file directory management nodes. In the method, after the main file directory management node is hung up, a new main file directory management node can be selected from the standby file directory management nodes within hundred milliseconds, and service is provided; and due to the consistency algorithm of the raft protocol, the data of the new main file directory management node is the same as the data of the hung main file directory management node, so that the problem that the standby file directory management node can take several minutes to provide service to the outside after the main file directory management node of the file storage system is hung is solved, and the problem that part of new metadata is lost after the main file directory management node of the file storage system is hung is also solved.
In the structure of the file storage system, the file directory management node for managing the directory tree is split into a plurality of file directory management fragments, and the file directory management fragments can be horizontally expanded in a seamless manner; the problem that a file storage system is limited by the memory bottleneck of a single file directory management node and can only store hundreds of millions of files is solved, and the quantity and the total data volume of the files stored by the system are increased.
The embodiment provides another data access method, wherein a plurality of file storage clusters are arranged in the file storage system; the metadata of the file to be accessed comprises a file storage cluster where the file data of the file to be accessed is located. Referring to the schematic structural diagram of the file storage system shown in fig. 2, a plurality of file storage clusters are arranged in the file storage system, and correspond to clusters in the diagram; the metadata of the file to be accessed includes an address, i.e., a cluster id, of a file storage cluster where the file data of the file to be accessed is located.
As shown in fig. 3, the method comprises the steps of:
step S302, acquiring the corresponding relation between the file directory management fragments and the data volumes through data persistence service;
for example, the data persistence service may be a Zookeeper (distributed application coordination service) service, and may be used to persist information such as IP (protocol for interconnection between networks) information, data volumes, and the like of the file directory management node, the cluster management node, and the data storage node in real time; data persistence service is usually set in a file storage system; the data persistence service is used for providing a corresponding relation between the file directory management fragments and the data volumes; the correspondence is usually obtained by persisting information of each file directory management node, cluster management node, and data storage node in real time. For example, when data of the file directory management node, the cluster management node, and the data storage node is changed, the changed data is persisted in real time, so as to ensure that the data in the data persistence service is the latest data.
Specifically, when the system is started, a server (corresponding to the Client in fig. 2) of the system accesses a primary data persistence service, obtains a corresponding relationship between file directory management fragments and data volumes from the data persistence service, and stores the corresponding relationship locally at the server; it should be noted that the server of the system only accesses the data persistence service once when the system is started, and stores the corresponding relationship between the required file directory management fragment and the data volume and other related information to the local; frequent access to data persistence services during data access is avoided.
Step S304, determining a first target data volume to which a file to be accessed belongs based on the data access request;
step S306, determining the file directory management fragment to which the first target data volume belongs based on the corresponding relation between the file directory management fragment and the data volume in the file storage system;
step S308, obtaining metadata of the file to be accessed from the file directory management fragment to which the first target data volume belongs;
step S310, the file to be accessed is obtained from the file storage cluster to which the file to be accessed belongs.
The metadata of the file to be accessed comprises the address of the file storage cluster where the file data of the file to be accessed is located, so that the corresponding file storage cluster can be directly accessed according to the address of the file storage cluster; and acquiring the file to be accessed through a cluster management node in the file storage cluster.
The file storage cluster comprises a cluster management node and a data storage node; the cluster management node stores the corresponding relation between the file data of each file stored in the file storage cluster and the data storage node. The cluster management node mainly functions to manage the mapping relationship from the metadata of the file to the file content (namely, the metadata of the file consists of several pieces of content, and each piece of content is distributed on several data storage nodes); the data storage node is used for storing file data uploaded by a user, and each file is horizontally cut into a plurality of blocks with fixed length.
Step S310, a possible implementation: determining a target data storage node where file data of a file to be accessed is located through a cluster management node of a file storage cluster to which the file to be accessed belongs; and acquiring file data of the file to be accessed from the target data storage node.
Specifically, a target data storage node corresponding to file data of a file to be accessed can be determined through a corresponding relation between the file data of each file in a master cluster management node of a file storage cluster to which the file to be accessed belongs and the data storage node; after the address of the target data storage node is obtained, the file data of the file to be accessed can be directly obtained from the target data storage node.
Each file storage cluster comprises a plurality of cluster management nodes; the plurality of cluster management nodes comprise a main cluster management node and at least one standby cluster management node; the main cluster management node is used for providing a target data storage node where file data of a file to be accessed is located; and the data of the standby cluster management node and the data of the main cluster management node are synchronized.
Specifically, refer to the schematic structural diagram of the file storage system shown in fig. 2; each file storage cluster usually comprises three or five cluster management nodes, wherein one cluster management node is a main cluster management node, and the rest cluster management nodes are standby cluster management nodes; the cluster management node corresponds to a MetaNode in FIG. 2; the main cluster management node corresponds to the Master MetaNode in FIG. 2; the standby cluster management node corresponds to the Slave MetaNode in FIG. 2; the data storage nodes correspond to the datanodes in fig. 2, and each file storage cluster may include a plurality of data storage nodes; the data of the plurality of cluster management nodes may be fully mirrored. The cluster management node mainly manages all data distribution in the file storage cluster and manages the mapping relation from the metadata of the file to the file content.
It should be noted that the file storage clusters included in the file storage system are not limited to the number shown in the drawings, but may be more, and the specific number is not limited herein, and may be set according to actual needs. In addition, the cluster management nodes are not limited to the number shown in the figure, and may also be more, and usually include three or five, where the number is not limited, and the specific number may be set according to actual needs; likewise, the number of data storage nodes is not limited to the number shown in the figure, and may be more or less, and is not limited herein.
In addition, a main cluster management node in the plurality of cluster management nodes is determined based on a preset raft protocol; and the data of the standby cluster management node is synchronized with the data of the main cluster management node through a preset raft protocol.
Usually, in an initial state, a plurality of cluster management nodes in each file storage cluster have no difference between active and standby, one active cluster management node is selected from the plurality of cluster management nodes through an election mechanism of a raft protocol, and the rest cluster management nodes directly become standby cluster management nodes. In the mode, after the main cluster management node is hung up, a new main cluster management node can be selected from the standby cluster management nodes within hundred milliseconds, and service is provided; due to the consistency algorithm of the raft protocol, the data of the new main cluster management node is the same as the data of the hung main cluster management node, the new main cluster management node is selected and served outside within hundred milliseconds after the main cluster management node of the file storage system is hung, the data of the new main cluster management node is completely consistent with the data of the old main cluster management node, and no data is lost. Meanwhile, the problem of storage capacity bottleneck caused by the fact that a file storage system is limited by a single cluster and cannot be horizontally expanded is solved.
In the structure of the file storage system, the cluster management node and the data storage node are split into a plurality of file storage clusters, and the file storage clusters can be horizontally and seamlessly expanded; the problem of storage capacity bottleneck caused by the fact that a file storage system is limited by a single cluster and cannot be expanded horizontally is solved, and the number of files stored in the system and the total data volume are increased.
Because the existing typical open source file storage system HDFS which is widely used has many defects, the most important of them is that the size of the stored files is limited, all the file metadata are stored in the NameNode node, and the NameNode has no horizontal capacity expansion capability, so that the number of files and the total data volume which can be stored by the cluster are limited by the memory bottleneck of the NameNode node, and generally only hundreds of millions of files can be stored, based on this, the embodiment of the present invention further provides a data storage method, as shown in fig. 4, the method includes the following steps:
step S402, determining a second target data volume to which the file to be stored belongs based on the data storage request;
the second target data volume may have the same attribute as the first target data volume, and may be a name space for storing directories, files, and the like created by the user.
Step S404, determining a file directory management fragment to which the second target data volume belongs based on the corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume;
step S406, determining a target storage position of a file to be stored; generating metadata of a file to be stored in a file directory management fragment to which a second target data volume belongs based on the target storage position; and storing the file to be stored to a target storage position.
The metadata may also be referred to as inode id, which is a unique identifier of the file to be stored, and may be defined as data describing the file, descriptive information about the file and information resources. The target storage position of the file to be stored can be determined by the file storage system, and the target storage position of the file to be stored is recorded at the same time, so that the subsequent user can conveniently access the file.
The file storage system is provided with a plurality of file storage clusters; the target storage location includes: a target file storage cluster storing file data of a file to be stored; the method for storing the file to be stored in the target storage position comprises the following steps: and storing the file to be stored to the target file storage cluster.
The file storage cluster comprises a cluster management node and a data storage node; the cluster management node stores the corresponding relation between the file data of each file stored in the file storage cluster and the data storage node; the method for storing the file to be stored in the target file storage cluster comprises the following steps:
1) determining a target data storage node for storing a file to be stored from a target file storage cluster;
specifically, a target data storage node for storing a file to be stored may be determined from cluster management nodes of the target file storage cluster.
2) Based on the target data storage node, generating a corresponding relation between file data of a file to be stored and the target data storage node in a cluster management node of a target file storage cluster; and storing the file data of the file to be stored to the target data storage node.
And generating the corresponding relation between the file data of the file to be stored and the target data storage node, so as to facilitate the access of subsequent users to the file data of the file to be stored.
In the above manner, based on the data storage request, the second target data volume to which the file to be stored belongs is determined; determining a file directory management fragment to which the second target data volume belongs based on a corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume; determining a target storage position of a file to be stored; generating metadata of a file to be stored in a file directory management fragment to which a second target data volume belongs based on the target storage position; and storing the file to be stored to a target storage position. In the mode, the file storage system is provided with the plurality of file directory management fragments, and when the file volume is large, the file target management fragments can be expanded horizontally, so that the problem that the number of metadata of the stored files in a single file directory management node is limited is solved, and the number of the stored files of the system and the total data volume are increased.
The present embodiment further provides a file storage system, as shown in fig. 5, the system includes a data request server, a plurality of file directory management fragments, and a plurality of file storage clusters; each file storage cluster is used for storing files in a specified data volume; each file directory management fragment is used for managing metadata of files in the designated data volume; the metadata is used for indicating a file storage cluster where the files in the specified data volume are located; and the data request server is used for managing the corresponding relation between the fragments and the data volumes based on the file directory and processing the received data request.
The file storage system is provided with a data persistence service; the data persistence service is used for providing the corresponding relation between the file directory management fragments and the data volumes to the data request server side.
Each file directory management fragment comprises a plurality of file directory management nodes; the file directory management nodes comprise a main file directory management node and at least one standby file directory management node; the main file directory management node is used for providing metadata of files in the designated data volume; and the standby file directory management node synchronizes the data of the main file directory management node.
The file storage cluster comprises a cluster management node and a data storage node; the data storage nodes are used for storing file data of each file in the file storage cluster; the cluster management node stores the corresponding relation between the file data of each file stored in the file storage cluster and the data storage node.
The file storage cluster comprises a plurality of cluster management nodes; the plurality of cluster management nodes comprise a main cluster management node and at least one standby cluster management node; the main cluster management node is used for providing a data storage node where file data of each file in the file storage cluster is located; and the data of the standby cluster management node and the data of the main cluster management node are synchronized.
The file storage system provided by this embodiment includes a data request server, a plurality of file directory management fragments, and a plurality of file storage clusters; each file storage cluster is used for storing files in a specified data volume; each file directory management fragment is used for managing metadata of files in the designated data volume; the metadata is used for indicating a file storage cluster where the files in the specified data volume are located; and the data request server is used for managing the corresponding relation between the fragments and the data volumes based on the file directory and processing the received data request. In the mode, the file storage system is provided with the plurality of file directory management fragments, and when the file volume is large, the file target management fragments can be expanded horizontally, so that the problem that the number of metadata of the stored files in a single file directory management node is limited is solved, and the number of the stored files of the system and the total data volume are increased.
Corresponding to the above method embodiment, this embodiment provides a data access apparatus, as shown in fig. 6, the apparatus includes:
the first data volume determining module 61 is configured to determine, based on the data access request, a first target data volume to which a file to be accessed belongs;
a first management fragment determining module 62, configured to determine a file directory management fragment to which the first target data volume belongs based on a correspondence between file directory management fragments and data volumes in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume;
the file access module 63 is configured to obtain metadata of a file to be accessed from the file directory management fragment to which the first target data volume belongs; and accessing the file to be accessed based on the metadata of the file to be accessed.
The embodiment of the invention provides a data access device, which is used for determining a first target data volume to which a file to be accessed belongs based on a data access request; determining a file directory management fragment to which the first target data volume belongs based on a corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume; acquiring metadata of a file to be accessed from a file directory management fragment to which a first target data volume belongs; and accessing the file to be accessed based on the metadata of the file to be accessed. In the mode, the file storage system is provided with the plurality of file directory management fragments, and when the file volume is large, the file target management fragments can be expanded horizontally, so that the problem that the number of metadata of the stored files in a single file directory management node is limited is solved, and the number of the stored files of the system and the total data volume are increased.
Further, each file directory management fragment comprises a plurality of file directory management nodes; the file directory management nodes comprise a main file directory management node and at least one standby file directory management node; the main file directory management node is used for providing metadata of a file to be accessed; and the standby file directory management node synchronizes the data of the main file directory management node.
Further, a main file directory management node in the plurality of file directory management nodes is determined based on a preset raft protocol; and the data of the standby file directory management node is synchronized with the data of the main file directory management node through a preset raft protocol.
Furthermore, a plurality of file storage clusters are arranged in the file storage system; the metadata of the file to be accessed comprises a file storage cluster where the file data of the file to be accessed is located; the file access module is further configured to: and acquiring the file to be accessed from the file storage cluster to which the file to be accessed belongs.
Further, the file storage cluster comprises a cluster management node and a data storage node; the cluster management node stores the corresponding relation between the file data of each file stored in the file storage cluster and the data storage node; the file access module is further configured to: determining a target data storage node where file data of a file to be accessed is located through a cluster management node of a file storage cluster to which the file to be accessed belongs; and acquiring file data of the file to be accessed from the target data storage node.
Further, each file storage cluster comprises a plurality of cluster management nodes; the plurality of cluster management nodes comprise a main cluster management node and at least one standby cluster management node; the main cluster management node is used for providing a target data storage node where file data of a file to be accessed is located; and the data of the standby cluster management node and the data of the main cluster management node are synchronized.
Further, a data persistence service is set in the file storage system; the data persistence service is used for providing a corresponding relation between the file directory management fragments and the data volumes; the device also comprises an acquisition module used for acquiring the corresponding relation between the file directory management fragments and the data volumes through the data persistence service.
The data access device provided by the embodiment of the invention has the same technical characteristics as the data access method provided by the embodiment, so that the same technical problems can be solved, and the same technical effects can be achieved.
Corresponding to the above method embodiment, this embodiment further provides a data storage device, as shown in fig. 7, where the data storage device includes:
the second data volume determining module 71 is configured to determine, based on the data storage request, a second target data volume to which the file to be stored belongs;
a second management fragment determining module 72, configured to determine a file directory management fragment to which the second target data volume belongs based on a correspondence between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume;
a file storage module 73, configured to determine a target storage location of a file to be stored; generating metadata of a file to be stored in a file directory management fragment to which a second target data volume belongs based on the target storage position; and storing the file to be stored to a target storage position.
In the device, a second target data volume to which a file to be stored belongs is determined based on a data storage request; determining a file directory management fragment to which the second target data volume belongs based on a corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in the designated data volume; determining a target storage position of a file to be stored; generating metadata of a file to be stored in a file directory management fragment to which a second target data volume belongs based on the target storage position; and storing the file to be stored to a target storage position. In the mode, the file storage system is provided with the plurality of file directory management fragments, and when the file volume is large, the file target management fragments can be expanded horizontally, so that the problem that the number of metadata of the stored files in a single file directory management node is limited is solved, and the number of the stored files of the system and the total data volume are increased.
Furthermore, a plurality of file storage clusters are arranged in the file storage system; the target storage location includes: a target file storage cluster storing file data of a file to be stored; the file storage module is further configured to: and storing the file to be stored to the target file storage cluster.
Further, the file storage cluster comprises a cluster management node and a data storage node; the cluster management node stores the corresponding relation between the file data of each file stored in the file storage cluster and the data storage node; the file storage module is further configured to: determining a target data storage node for storing a file to be stored from a target file storage cluster; based on the target data storage node, generating a corresponding relation between file data of a file to be stored and the target data storage node in a cluster management node of a target file storage cluster; and storing the file data of the file to be stored to the target data storage node.
The data storage device provided by the embodiment of the invention has the same technical characteristics as the data storage method provided by the embodiment, so that the same technical problems can be solved, and the same technical effects can be achieved.
The embodiment also provides an electronic device, which comprises a processor and a memory, wherein the memory stores machine executable instructions capable of being executed by the processor, and the processor executes the machine executable instructions to realize the data access party, the data storage method or the file storage system.
Referring to fig. 8, the electronic device includes a processor 100 and a memory 101, the memory 101 stores machine executable instructions capable of being executed by the processor 100, and the processor 100 executes the machine executable instructions to implement the data access method, the data storage method or the file storage system.
Further, the electronic device shown in fig. 8 further includes a bus 102 and a communication interface 103, and the processor 100, the communication interface 103, and the memory 101 are connected through the bus 102.
The Memory 101 may include a high-speed Random Access Memory (RAM) and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used. The bus 102 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 8, but that does not indicate only one bus or one type of bus.
The present embodiments also provide a machine-readable storage medium having stored thereon machine-executable instructions that, when invoked and executed by a processor, cause the processor to implement the data access method, data storage method or file storage system described above.
The data access method, the data storage device and the computer program product of the file storage system provided by the embodiments of the present invention include a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the method described in the foregoing method embodiments, and specific implementation may refer to the method embodiments, and will not be described herein again.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the system and the apparatus described above may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present invention can be understood in specific cases for those skilled in the art.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that the following embodiments are merely illustrative of the present invention, and not restrictive, and the scope of the present invention is not limited thereto: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (19)
1. A method of data access, the method comprising:
determining a first target data volume to which a file to be accessed belongs based on the data access request;
determining a file directory management fragment to which the first target data volume belongs based on a corresponding relation between the file directory management fragment and the data volume in a file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in a specified data volume;
acquiring metadata of the file to be accessed from the file directory management fragment to which the first target data volume belongs; and accessing the file to be accessed based on the metadata of the file to be accessed.
2. The method of claim 1, wherein each of the file directory management slices comprises a plurality of file directory management nodes; the file directory management nodes comprise a main file directory management node and at least one standby file directory management node;
the main file directory management node is used for providing metadata of the file to be accessed; and the data of the standby file directory management node and the data of the main file directory management node are synchronized.
3. The method according to claim 2, wherein a primary file directory management node among the plurality of file directory management nodes is determined based on a preset raft protocol;
and the data of the standby file directory management node is synchronized with the data of the main file directory management node through a preset raft protocol.
4. The method of claim 1, wherein a plurality of file storage clusters are disposed in the file storage system; the metadata of the file to be accessed comprises a file storage cluster where the file data of the file to be accessed is located;
the step of accessing the file to be accessed based on the metadata of the file to be accessed comprises: and acquiring the file to be accessed from the file storage cluster to which the file to be accessed belongs.
5. The method of claim 4, wherein the file storage cluster comprises a cluster management node and a data storage node; the cluster management node stores the corresponding relation between the file data of each file stored in the file storage cluster and the data storage node;
the step of obtaining the file to be accessed from the file storage cluster to which the file to be accessed belongs comprises:
determining a target data storage node where file data of the file to be accessed is located through a cluster management node of a file storage cluster to which the file to be accessed belongs;
and acquiring the file data of the file to be accessed from the target data storage node.
6. The method of claim 5, wherein each of the file storage clusters comprises a plurality of cluster management nodes; the cluster management nodes comprise a main cluster management node and at least one standby cluster management node;
the main cluster management node is used for providing a target data storage node where the file data of the file to be accessed is located; and the data of the standby cluster management node and the data of the main cluster management node are synchronous.
7. The method according to claim 1, wherein a data persistence service is provided in the file storage system; the data persistence service is used for providing the corresponding relation between the file directory management fragments and the data volumes;
before the step of determining, based on the data access request, a first target data volume to which the file to be accessed belongs, the method further includes: and acquiring the corresponding relation between the file directory management fragments and the data volumes through the data persistence service.
8. A method of data storage, the method comprising:
determining a second target data volume to which the file to be stored belongs based on the data storage request;
determining a file directory management fragment to which the second target data volume belongs based on a corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in a specified data volume;
determining a target storage position of the file to be stored; generating metadata of the file to be stored in the file directory management fragment to which the second target data volume belongs based on the target storage position; and storing the file to be stored to the target storage position.
9. The method of claim 8, wherein a plurality of file storage clusters are disposed in the file storage system; the target storage location includes: a target file storage cluster for storing the file data of the file to be stored;
the step of storing the file to be stored to the target storage location includes: and storing the file to be stored to the target file storage cluster.
10. The method of claim 9, wherein the file storage cluster comprises a cluster management node and a data storage node; the cluster management node stores the corresponding relation between the file data of each file stored in the file storage cluster and the data storage node;
the step of storing the file to be stored in the target file storage cluster includes:
determining a target data storage node for storing the file to be stored from the target file storage cluster;
generating a corresponding relation between the file data of the file to be stored and the target data storage node in a cluster management node of the target file storage cluster based on the target data storage node; and storing the file data of the file to be stored to the target data storage node.
11. A file storage system is characterized by comprising a data request server, a plurality of file directory management fragments and a plurality of file storage clusters;
wherein each file storage cluster is used for storing files in a designated data volume; each file directory management fragment is used for managing metadata of files in a specified data volume; the metadata is used for indicating a file storage cluster where the files in the specified data volume are located;
and the data request server is used for processing the received data request based on the corresponding relation between the file directory management fragments and the data volumes.
12. The system according to claim 11, wherein a data persistence service is provided in the file storage system; the data persistence service is used for providing the corresponding relation between the file directory management fragments and the data volumes to the data request server side.
13. The system according to claim 11, wherein each of said file directory management slices comprises a plurality of file directory management nodes; the file directory management nodes comprise a main file directory management node and at least one standby file directory management node;
the main file directory management node is used for providing metadata of files in the specified data volume; and the data of the standby file directory management node and the data of the main file directory management node are synchronized.
14. The system of claim 11, wherein the file storage cluster comprises a cluster management node and a data storage node;
the data storage nodes are used for storing file data of each file in the file storage cluster; the cluster management node stores the corresponding relation between the file data of each file stored in the file storage cluster and the data storage node.
15. The system of claim 14, wherein the file storage cluster comprises a plurality of cluster management nodes; the cluster management nodes comprise a main cluster management node and at least one standby cluster management node;
the main cluster management node is used for providing a data storage node where file data of each file in the file storage cluster is located; and the data of the standby cluster management node and the data of the main cluster management node are synchronous.
16. A data access apparatus, characterized in that the apparatus comprises:
the first data volume determining module is used for determining a first target data volume to which a file to be accessed belongs based on the data access request;
the first management fragment determining module is used for determining the file directory management fragment to which the first target data volume belongs based on the corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in a specified data volume;
the file access module is used for acquiring metadata of the file to be accessed from the file directory management fragment to which the first target data volume belongs; and accessing the file to be accessed based on the metadata of the file to be accessed.
17. A data storage device, characterized in that the device comprises:
the second data volume determining module is used for determining a second target data volume to which the file to be stored belongs based on the data storage request;
the second management fragment determining module is used for determining the file directory management fragment to which the second target data volume belongs based on the corresponding relation between the file directory management fragment and the data volume in the file storage system; the file storage system is provided with a plurality of file directory management fragments; each file directory management fragment is used for managing metadata of files in a specified data volume;
the file storage module is used for determining a target storage position of the file to be stored; generating metadata of the file to be stored in the file directory management fragment to which the second target data volume belongs based on the target storage position; and storing the file to be stored to the target storage position.
18. An electronic device comprising a processor and a memory, the memory storing machine executable instructions executable by the processor to perform the data access method of any one of claims 1 to 7, the data storage method of any one of claims 8 to 10, or the file storage system of any one of claims 11 to 15.
19. A machine-readable storage medium having stored thereon machine-executable instructions which, when invoked and executed by a processor, cause the processor to implement the data access method of any of claims 1 to 7, the data storage method of any of claims 8 to 10, or the file storage system of any of claims 11 to 15.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010965297.0A CN112100129A (en) | 2020-09-14 | 2020-09-14 | Data access method, data storage method, data access device and file storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010965297.0A CN112100129A (en) | 2020-09-14 | 2020-09-14 | Data access method, data storage method, data access device and file storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112100129A true CN112100129A (en) | 2020-12-18 |
Family
ID=73759912
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010965297.0A Pending CN112100129A (en) | 2020-09-14 | 2020-09-14 | Data access method, data storage method, data access device and file storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112100129A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100115011A1 (en) * | 2008-10-30 | 2010-05-06 | Callahan Michael J | Enumerating Metadata in File System Directories |
| CN102521386A (en) * | 2011-12-22 | 2012-06-27 | 清华大学 | Method for grouping space metadata based on cluster storage |
| CN103647797A (en) * | 2013-11-15 | 2014-03-19 | 北京邮电大学 | Distributed file system and data access method thereof |
| US20140108723A1 (en) * | 2012-10-17 | 2014-04-17 | Datadirect Networks, Inc. | Reducing metadata in a write-anywhere storage system |
| CN107315547A (en) * | 2017-07-18 | 2017-11-03 | 郑州云海信息技术有限公司 | A kind of method and device for reading distributed meta data file |
| CN107357681A (en) * | 2017-06-26 | 2017-11-17 | 杭州铭师堂教育科技发展有限公司 | Zookeeper backup management systems and method based on salt |
| CN107547653A (en) * | 2017-09-11 | 2018-01-05 | 华北水利水电大学 | A kind of distributed file storage system |
| US10140304B1 (en) * | 2015-12-10 | 2018-11-27 | EMC IP Holding Company LLC | Distributed metadata servers in a file system with separate metadata servers for file metadata and directory metadata |
| US20190361988A1 (en) * | 2018-05-25 | 2019-11-28 | Microsoft Technology Licensing, Llc | Scalable multi-tier storage structures and techniques for accessing entries therein |
-
2020
- 2020-09-14 CN CN202010965297.0A patent/CN112100129A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100115011A1 (en) * | 2008-10-30 | 2010-05-06 | Callahan Michael J | Enumerating Metadata in File System Directories |
| CN102521386A (en) * | 2011-12-22 | 2012-06-27 | 清华大学 | Method for grouping space metadata based on cluster storage |
| US20140108723A1 (en) * | 2012-10-17 | 2014-04-17 | Datadirect Networks, Inc. | Reducing metadata in a write-anywhere storage system |
| CN103647797A (en) * | 2013-11-15 | 2014-03-19 | 北京邮电大学 | Distributed file system and data access method thereof |
| US10140304B1 (en) * | 2015-12-10 | 2018-11-27 | EMC IP Holding Company LLC | Distributed metadata servers in a file system with separate metadata servers for file metadata and directory metadata |
| CN107357681A (en) * | 2017-06-26 | 2017-11-17 | 杭州铭师堂教育科技发展有限公司 | Zookeeper backup management systems and method based on salt |
| CN107315547A (en) * | 2017-07-18 | 2017-11-03 | 郑州云海信息技术有限公司 | A kind of method and device for reading distributed meta data file |
| CN107547653A (en) * | 2017-09-11 | 2018-01-05 | 华北水利水电大学 | A kind of distributed file storage system |
| US20190361988A1 (en) * | 2018-05-25 | 2019-11-28 | Microsoft Technology Licensing, Llc | Scalable multi-tier storage structures and techniques for accessing entries therein |
Non-Patent Citations (1)
| Title |
|---|
| 马灿等: "曙光星云分布式文件系统: 海量小文件存取", 《小型微型计算机系统》, vol. 33, no. 7, 31 July 2012 (2012-07-31), pages 1481 - 1488 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11379461B2 (en) | Multi-master architectures for distributed databases | |
| CN109783438B (en) | Distributed NFS system based on librados and construction method thereof | |
| US8775373B1 (en) | Deleting content in a distributed computing environment | |
| US9560093B2 (en) | Higher efficiency storage replication using compression | |
| US8346824B1 (en) | Data distribution system | |
| US10579615B2 (en) | Method for data retrieval from a distributed data storage system | |
| EP2498476B1 (en) | Massively scalable object storage system | |
| CN112199419B (en) | Distributed time sequence database, storage method, equipment and storage medium | |
| CN111049928B (en) | Data synchronization method, system, electronic device and computer readable storage medium | |
| CN113360456B (en) | Data archiving method, device, equipment and storage medium | |
| CN112632029B (en) | Data management method, device and equipment of distributed storage system | |
| CN111259072A (en) | Data synchronization method and device, electronic equipment and computer readable storage medium | |
| US8417679B1 (en) | Fast storage writes | |
| CN112000850A (en) | Method, device, system and equipment for data processing | |
| CN112100129A (en) | Data access method, data storage method, data access device and file storage system | |
| CN113296714B (en) | Data storage system based on NAS protocol | |
| CN116737764A (en) | Method and device for data synchronization, electronic equipment and storage medium | |
| CN111104376B (en) | Resource file query method and device | |
| CN112291343A (en) | Information acquisition method and device and electronic equipment | |
| CN113760519A (en) | Distributed transaction processing method, device and system and electronic equipment | |
| CN111352916A (en) | Data storage method, system and storage medium based on NAS storage system | |
| CN112698935A (en) | Data access method, data server and data storage system | |
| CN117742571A (en) | Data processing method, device, equipment and readable storage medium | |
| CN113836225A (en) | Object storage multi-site synchronous alarm method, system, terminal and storage medium | |
| CN117424907A (en) | File acquisition method, device, node equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |