CN106790434B - Network data management method, network attached storage gateway and storage service system - Google Patents
Network data management method, network attached storage gateway and storage service system Download PDFInfo
- Publication number
- CN106790434B CN106790434B CN201611103290.8A CN201611103290A CN106790434B CN 106790434 B CN106790434 B CN 106790434B CN 201611103290 A CN201611103290 A CN 201611103290A CN 106790434 B CN106790434 B CN 106790434B
- Authority
- CN
- China
- Prior art keywords
- network
- network data
- file
- data
- attached storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/66—Arrangements for connecting between networks having differing types of switching systems, e.g. gateways
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a network data management method, a network attached storage gateway and a storage service system, wherein the network data management method is applied to the storage service system comprising a plurality of groups of clients, a plurality of network attached storage gateway groups and a distributed object file system cluster; each group of network attached storage gateway groups comprises a plurality of network attached storage gateways, the client is connected with the network attached storage gateway groups, and the plurality of network attached storage gateway groups are connected with the distributed object file system cluster; the distributed object file system cluster is provided with a distributed object file system storing network data, the distributed object file system maps the stored network data to a network attached storage gateway so as to enable the network attached storage gateway to cache the network data, and the management method comprises the following steps: receiving an access request of network data; a network data management operation associated with the request for access to the network data is performed. The invention provides a high-performance network file system while ensuring high expansibility and high reliability.
Description
Technical Field
The invention belongs to the technical field of computer storage, relates to a management method and a management system, and particularly relates to a network data management method, a network attached storage gateway and a storage service system.
Background
Conventional Network Attached Storage (NAS) can meet general file storage and management requirements, but makes a significant compromise in terms of reliability and scalability of storage.
And the distributed object file system based on the object storage provides better expandability and reliability. However, the access delay of the distributed object file system is high, random writing is not supported, network file service cannot be directly provided for the client computer, the distributed object file system is generally provided for application programs through Restful API, and it is very difficult to readjust the storage interface for a large number of business systems. However, the simple addition of NAS storage protocols to the distributed object file system tends to result in poor performance of adapted NAS storage due to the difference between access and storage modes.
Therefore, how to provide a network data management method, a network attached storage gateway and a storage service system to solve the defects that the access performance to a distributed object file system is low and various network data cannot be uniformly stored and managed because a high-performance gateway which can be directly mounted and used by a client cannot be established on the distributed object file system in the prior art, and the like, is really a technical problem to be solved by practitioners in the art.
Disclosure of Invention
In view of the above drawbacks of the prior art, an object of the present invention is to provide a method for managing network data, a network attached storage gateway, and a storage service system, which are used to solve the problems that in the prior art, a high-performance gateway that can be directly mounted and used by a client cannot be established on a distributed object file system, so that the access performance to the distributed object file system is low, and unified storage and management of various types of network data cannot be achieved.
In order to achieve the above and other related objects, an aspect of the present invention provides a method for managing network data, which is applied to a network data storage service system including a plurality of groups of clients, a plurality of network attached storage gateway groups, and a distributed object file system cluster; each group of network attached storage gateway groups comprises a plurality of network attached storage gateways, each group of clients is connected with one network attached storage gateway group, and the plurality of network attached storage gateway groups are connected with the distributed object file system cluster; the distributed object file system cluster is used for mapping the stored network data to the network attached storage gateways in the network attached storage gateway group so as to cache the network data, and the management method of the network data comprises the following steps: receiving an access request of network data sent by the client bound with the access request; and according to the access request of the network data, executing network data management operation associated with the access request of the network data.
In an embodiment of the present invention, the network data includes metadata and file data; the request for access to the network data comprises: a read request of file data in the network data, a write request of file data in the network data, and/or an operation request of metadata in the network data.
In an embodiment of the present invention, the distributed object file system cluster includes a metadata storage module for storing metadata in network data and an object storage module for storing file data in network data; each network attached storage gateway comprises a metadata cache unit for caching metadata in network data, a file data cache unit for caching file data in the network data, and a high-availability management unit which is bound with different virtual IP addresses and detects heartbeat of each network attached storage gateway in the group of network attached storage gateways.
In an embodiment of the present invention, the network data read operation associated with the read request of the file data in the network data includes: after receiving the reading request of the network data, judging whether the network attached storage gateway caches the file data in the network data to be read after sending the reading request; if the file data is cached, reading the file data from a data caching unit of the file data of the network attached storage gateway, and feeding the read file data back to the client; if the network data is not cached, the reading request of the network data is forwarded to the distributed object file system cluster, the network data is obtained and downloaded through random access, the downloaded network data is cached, and the file data of the network data is fed back to the client.
In an embodiment of the present invention, the network data write operation associated with the write request of the file data in the network data includes: after receiving the writing request of the network data, opening a file data cache unit of the network attached storage gateway, and randomly writing the file data of the network data into the file data cache unit to cache the written file data until the network data is completely written; after the network data are completely written, closing a file data cache unit of the network attached storage gateway, and uploading the written file data to the distributed object file system; and simultaneously feeding back the information that the file data caching unit is closed to the client.
In an embodiment of the present invention, the file/directory operation of the metadata in the network data associated with the operation request of the metadata in the network data includes: after receiving a file/directory operation request of metadata in the network data, executing file/directory operation, and feeding back an operation result of the file/directory operation to the client; wherein the file/directory operation request of the metadata in the network data comprises a file/directory read operation request of the metadata in the network data and a file/directory update request of the metadata in the network data; the file/directory operation of the network data comprises a file/directory read operation of metadata in the network data and a file/directory modification operation of the metadata in the network data.
In an embodiment of the present invention, the file/directory reading operation of the network data includes: after receiving a file/directory reading operation request of metadata in the network data, directly reading the metadata from a metadata cache unit of the network attached storage gateway, and feeding back the read metadata to a client; and periodically reading the metadata which is not cached in the metadata caching unit from the metadata storage module of the distributed object file system so as to update the cached metadata.
In an embodiment of the present invention, the file/directory modification operation of the network data includes: when the metadata in the metadata cache unit of the network attached storage gateway is updated, the updated metadata is written into the metadata storage module of the distributed object file system, and the metadata in the metadata cache unit of the network attached storage gateway is fed back to the client side as an updated operation result.
The invention provides a network attached storage gateway, which is applied to a network data storage service system comprising a plurality of groups of clients, a plurality of network attached storage gateway groups and a distributed object file system cluster; each group of network attached storage gateway groups comprises a plurality of network attached storage gateways, each group of clients is connected with one network attached storage gateway group, and the plurality of network attached storage gateway groups are connected with the distributed object file system cluster; wherein the distributed object file system cluster stores network data, the distributed object file system cluster maps the stored network data to the network attached storage gateways in the network attached storage gateway group to enable the network attached storage gateways to cache the network data, and the network attached storage gateways comprise: the metadata caching unit is used for caching metadata in the network data; the object cache unit is used for caching the file data in the network data; the operation unit is used for receiving an access request of the network data sent by the client side and bound with the operation unit; and according to the access request of the network data, executing network data management operation associated with the access request of the network data.
The invention also provides a network data storage service system, which comprises a plurality of groups of clients, a plurality of network attached storage gateway groups and a network data storage service system of a distributed object file system cluster; each group of network attached storage gateway groups comprises a plurality of network attached storage gateways, each group of clients is connected with one network attached storage gateway group, and the plurality of network attached storage gateway groups are connected with the distributed object file system cluster; the distributed object file system cluster maps the stored network data to the network attached storage gateways in the network attached storage gateway group so as to cache the network data.
As described above, the network data management method, the network attached storage gateway, and the storage service system according to the present invention have the following advantageous effects:
according to the storage service system of the network data, the management method of the network data based on the storage service system and the network attached storage gateway, when the NAS gateway receives an access request of the network data, the NAS gateway does not directly access the data of the distributed object file system, and the access performance of a client is improved through a high-performance cache and a metadata cache. Therefore, the invention effectively realizes that the distributed object file system is mapped into the NAS storage, ensures high reliability and high expandability of the storage, and realizes a high-performance and standard file storage interface of the NAS storage, so that the network file system has high performance, reliability and expandability.
Drawings
Fig. 1 is a schematic structural diagram of a network data storage service system according to an embodiment of the present invention.
Fig. 2 is a flowchart illustrating a network data management method according to an embodiment of the invention.
FIG. 3 is a timing flow diagram illustrating network data management operations associated with a read request for file data in network data according to the present invention.
FIG. 4 is a timing flow diagram illustrating network data management operations associated with a request to write file data in network data according to the present invention.
FIG. 5A is a flowchart illustrating a file/directory read operation performed as metadata in network data according to the present invention.
FIG. 5B is a flowchart illustrating a file/directory modification operation performed as metadata in network data according to the present invention.
FIG. 6 is a schematic diagram of a network attached storage network according to an embodiment of the present invention.
Description of the element reference numerals
1 storage service system of network data
11 client group
12 network attached storage gateway group
13 distributed object file system cluster
111A, client
111B、
111C、
112A、
112B、
113A、
113B、
114A、
114B、
114C
121. Network attached storage gateway
122、
123、
124
13A metadata storage module
13B object storage module
13C metadata repository
21 metadata cache unit
22 file data buffer unit
23 operating unit
24 high availability management unit
S1-S10
Detailed Description
The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention, and the components related to the present invention are only shown in the drawings rather than drawn according to the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.
Examples
Fig. 1 shows a schematic structural diagram of a network data storage service system in an embodiment. As shown in fig. 1, the storage service system 1 for network data includes a plurality of groups of clients 11, a plurality of network attached storage gateway (NAS) groups 12, and a distributed object file system cluster 13. In this embodiment, each group of clients includes a client 1 group, a client 2 group, a client 3 group, and a client 4 group, the client group 11 includes clients 111A, 111B, and 111C, the client 2 group includes clients 112A and 112B, the client 3 group includes clients 113A and 113B, and the client 4 group includes clients 114A, 114B, and 114C. The network attached storage gateway group includes network attached storage gateway 12 including a number of network attached storage gateways 121, 122, 123, 124. Each group of clients 11 is connected to one network attached storage gateway group 12, i.e. the clients 111A, 111B, 111C are connected to the network attached storage gateway 121, the clients 112A and 112B are connected to the network attached storage gateway 122, and a plurality of network attached storage gateway groups 12, i.e. the network attached storage gateways 121, 122, 123, 124 are connected to the distributed object file system cluster 13. In this embodiment, the distributed object file system cluster 13 stores network data, and the distributed object file system cluster 13 maps the stored network data to the network attached storage gateways 121, 122, 123, and 124 in the network attached storage gateway group 12, so that the network data is cached by the network attached storage gateways. In this embodiment, the network data includes metadata and file data. The distributed object file system cluster 1313 includes a metadata storage module 13A for storing metadata in network data, an object storage module 13B for storing file data in network data, and a metadata repository 13C connected to the metadata storage module 13A for storing all metadata; the metadata storage module 13A, the object storage module 13B, and the metadata database 13C are disposed in different servers. Thus, the distributed object file system cluster 13 is also made up of different servers. The metadata includes a data object ID, a data object name, a data object type, a storage name, a storage service access address, a storage access ID (access ID), a storage access value (access key), a path of the data object in storage, a data object size, a data object etag (hash value of the object), a data object state, a data object creation time, a data object final modification time, and the like. The storage service system 1 for network data stores network data by using object storage. Based on the distributed object storage cluster, Restful API can be provided, and the upper layer can realize application services such as cloud disk and document management.
In this embodiment, the network attached storage gateway implements network file system access protocols, including CIFS/SMB, NFS, FTP, WebDAV, NDMP, and other protocols. The NAS gateway processes an access request of a file from a client computer, integrates random read-write access and converts the random read-write access into access with an object storage and taking a data object as a unit; the cache management is provided, the access performance of object storage is improved, and the high availability of the dual-active gateway is realized through write cache synchronization.
Each of the network attached storage gateways 121, 122, 123, 124 includes a metadata cache unit 12A for caching metadata in network data, a file data cache unit 12B for caching file data in the network data, and a high availability management unit (not shown) for binding with different virtual IP addresses and detecting heartbeat of each network attached storage gateway in the set of network attached storage gateways. The high-availability management unit detects heartbeat signals between the two gateways, when one NAS gateway is offline, the other NAS gateway cannot receive the heartbeat signals of the NAS of the opposite side, and the other NAS gateway takes over the VIP of the NAS gateway to continuously provide file storage service for the client side.
The storage service system 1 based on the network data will be described in detail below with reference to the drawings. Please refer to fig. 2, which is a flowchart illustrating a network data management method according to an embodiment of the present invention. As shown in fig. 2, the method for managing network data specifically includes the following steps:
s1, the network attached storage gateway group receives the access request of the network data sent by the client bound with the network attached storage gateway group. In this embodiment, the network data includes metadata and file data; the request for access to the network data comprises: a read request of file data in the network data, a write request of file data in the network data, and/or an operation request of metadata in the network data.
And S2, according to the access request of the network data, executing the network data management operation associated with the access request of the network data.
In this embodiment, when the access request of the network data is a read request of file data in the network data, please refer to fig. 3, which is a timing flow chart illustrating a network data management operation associated with the read request of file data in the network data. As shown in fig. 3, the network data read operation associated with the access request of the network data includes:
and S3, after receiving the reading request of the file data in the network data sent by the client, the NAS gateway judges whether the network attached storage gateway caches the file data in the network data to be read by sending the reading request.
And S4, if the file data is cached, reading the file data from the file data caching unit of the network attached storage gateway, and feeding the read Chinese file data back to the client. In this embodiment, the data object content storage portion of the distributed object file system cluster supports random reading of data object content, which can avoid the waiting time of the first read request. And the metadata cache unit of the NAS gateway needs to realize cache management, the cache space is configurable, and the cache elimination algorithm can adopt algorithms such as LRU (least recently used) and the like.
And S5, if the network data is not cached, forwarding the reading request of the file data in the network data to the object storage module of the distributed object file system cluster, acquiring and downloading the file data in the network data through random access, caching the downloaded file data in the network data, and feeding back the downloaded file data of the network data to the client.
In this embodiment, when the access request of the network data is a write request of file data in the network data, please refer to fig. 4, which is a timing flow chart illustrating a network data management operation associated with the write request of the file data in the network data. As shown in fig. 4, the network data write operation associated with the write request of the file data in the network data includes:
s6, after receiving the writing request of the file data in the network data sent by the client, the NAS gateway opens the file data cache unit of the network attached storage gateway, and writes the file data of the network data into the file data cache unit randomly to cache the written metadata into the file data cache unit until the network data is completely written.
S7, when the network data is completely written, the client requests the NAS gateway to close the file data cache unit so as to close the file data cache unit of the network attached storage gateway, and the written file data is uploaded to the object storage module of the distributed object file system cluster; and simultaneously feeding back the information that the file data caching unit is closed to the client.
In this embodiment, when writing the file data in the network data into the file data caching unit of the network attached storage gateway, the method further includes determining the cache space capacity of the data caching unit of the file data, when the cache space capacity reaches a preset upper limit value, sorting the cache space, eliminating part of the file data of the network data cached, and releasing the cache space.
In this embodiment, after receiving a file/directory operation request of metadata in the network data, the file/directory operation is executed, and an operation result of the file/directory operation is fed back to the client; wherein the file/directory operation request of the metadata in the network data comprises a file/directory read operation request of the metadata in the network data and a file/directory update request of the metadata in the network data; the file/directory operation of the metadata in the network data comprises a file/directory read operation of the metadata in the network data and a file/directory modification operation of the metadata in the network data.
The file/directory operation request of the metadata in the network data comprises a file/directory read operation request of the metadata in the network data and a file/directory update request of the metadata in the network data; the file/directory operation of the metadata in the network data comprises a file/directory read operation of the metadata in the network data and a file/directory modification operation of the metadata in the network data.
When the access request of the network data is a file/directory read operation request of metadata in the network data, please refer to fig. 5A, which is a flowchart illustrating a file/directory read operation timing of metadata in the network data. As shown in fig. 5A, the file/directory read operation of the metadata in the network data includes:
s8, after receiving the file/directory reading operation request of the metadata in the network data, directly reading the metadata from the metadata cache unit of the network attached storage gateway, and feeding back the read metadata to the client;
s9, periodically reading the metadata which is not cached in the metadata cache unit from the metadata storage module of the distributed object file system cluster to update the cached metadata. In this embodiment, the updating of the metadata cache is based on the updating of the state consistency, so that when the metadata cache is read, whether to immediately start the updating of the metadata cache of the specified file or directory can be determined by judging the consistency state.
When the access request of the network data is a file/directory modification operation request of metadata in the network data, please refer to fig. 5B, which is a flowchart illustrating a file/directory modification operation timing sequence of metadata in the network data. As shown in fig. 5B, the file/directory modification operation of the metadata in the network data includes:
and S10, when updating the metadata in the metadata cache unit of the network attached storage gateway, writing the updated metadata into the metadata storage module of the distributed object file system cluster, and feeding back the metadata in the metadata cache unit of the network attached storage gateway to the client side with the updated operation result.
In the storage service system of network data and the management method of network data based on the storage service system described in this embodiment, when the NAS gateway receives an access request of network data, the NAS gateway does not directly access cluster data of the distributed object file system, but improves access performance of a client through a high performance cache and a metadata cache. Therefore, the invention effectively realizes the mapping of the cluster of the distributed object file system into the NAS storage, ensures the high reliability and the high expandability of the storage, and realizes the high-performance and standard file storage interface of the NAS storage, so that the network file system has high performance, reliability and expandability.
Fig. 6 is a schematic diagram of a schematic structure of a network attached storage network according to an embodiment. As shown in fig. 6, specifically, the network attached storage gateway 2 includes: a metadata cache unit 21, a file data cache unit 22, an operation unit 23, and a high availability management unit 24.
The metadata caching unit 21 is used for caching metadata in the network data.
The file data caching unit 22 is used for caching file data in the network data.
The operation unit 23 connected to the metadata cache unit 21 and the file data cache unit 22 is configured to receive an access request of network data sent by the client bound to the operation unit; and according to the access request of the network data, executing network data management operation associated with the access request of the network data. That is, the operation unit 23 is configured to respond to a metadata query and update operation of a response client, and update and synchronize with a metadata module of the distributed object file system cluster; and responding to the file data reading and updating writing operations of the client, and synchronizing and updating the file data stored in the distributed object file system cluster.
Specifically, after receiving a read request of file data in the network data, the operation unit 23 is configured to determine whether the network attached storage gateway has cached the file data in the network data to be read by sending the read request; if the file data is cached, reading the file data from a file data caching unit of the network attached storage gateway, and feeding the read file data back to the client; and if the network data is not cached, forwarding a reading request of the file data in the network data to an object storage module of the distributed object file system cluster, acquiring and downloading the file data in the network data through random access, caching the downloaded file data in the network data, and feeding back the file data in the network data to the client.
After receiving a write-in request of file data in the network data, the operation unit 23 connected to the metadata cache unit 21 and the object cache unit 22 is configured to open a file data cache unit of the network attached storage gateway, and randomly write the file data of the network data into the file data cache unit to cache the written-in file data until all the file data in the network data are written; after the network data file data is completely written, closing a file data cache unit of the network attached storage gateway, and uploading the written file data to the distributed object file system cluster; and simultaneously feeding back the information that the file data caching unit is closed to the client.
After receiving a file/directory operation request of metadata in the network data, the operation module 23 executes a file/directory operation and feeds back an operation result of the file/directory operation to the client; wherein the file/directory operation request of the metadata in the network data comprises a file/directory read operation request of the metadata in the network data and a file/directory update request of the metadata in the network data; the file/directory operation of the metadata in the network data comprises a file/directory read operation of the metadata in the network data and a file/directory modification operation of the metadata in the network data.
After receiving a file/directory reading operation request of metadata in the network data, the operation unit 23 is configured to directly read the metadata from the metadata cache unit of the network attached storage gateway, and feed back the read metadata to the client; and periodically reading the metadata which is not cached in the metadata caching unit from the metadata storage module of the distributed object file system cluster so as to update the cached metadata. When updating the metadata in the metadata cache unit of the network attached storage gateway, the operation unit 23 is configured to write the updated metadata into the metadata storage module of the distributed object file system cluster, and simultaneously feed back the metadata in the metadata cache unit of the network attached storage gateway to the client as an updated operation result.
The high availability management unit 24 connected to the operation unit 23 is configured to bind with different virtual IP addresses and detect a heartbeat of each network attached storage gateway in the set of network attached storage gateways.
In summary, according to the storage service system for network data, the management method for network data based on the storage service system, and the network attached storage gateway provided by the present invention, when the NAS gateway receives an access request for network data, the NAS gateway does not directly access data of the distributed object file system cluster, but improves the access performance of the client through the high performance cache and the metadata cache. Therefore, the invention effectively realizes the mapping of the cluster of the distributed object file system into the NAS storage, ensures the high reliability and the high expandability of the storage, and realizes the high-performance and standard file storage interface of the NAS storage, so that the network file system has high performance, reliability and expandability. Therefore, the invention effectively overcomes various defects in the prior art and has high industrial utilization value.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.
Claims (10)
1. A management method of network data is characterized in that the management method is applied to a network data storage service system comprising a plurality of groups of clients, a plurality of network attached storage gateway groups and a distributed object file system cluster; each group of network attached storage gateway groups comprises a plurality of network attached storage gateways, each group of clients is connected with one network attached storage gateway group, and the plurality of network attached storage gateway groups are connected with the distributed object file system cluster; the distributed object file system cluster is used for mapping the stored network data to the network attached storage gateways in the network attached storage gateway group so as to cache the network data; each network attached storage gateway comprises a metadata cache unit for caching metadata in network data, a file data cache unit for caching file data in the network data, and a high-availability management unit which is bound with different virtual IP addresses and detects heartbeat of each network attached storage gateway in the group of network attached storage gateways; the high-availability management unit detects heartbeat signals between the two gateways, when one network attached storage gateway is offline, the other network attached storage gateway cannot receive the heartbeat signals of the other network attached storage gateway, and the other network attached storage gateway takes over the VIP of the network attached storage gateway to continuously provide file storage service for the client; the management method of the network data comprises the following steps:
receiving an access request of network data sent by the client bound with the access request;
according to the access request of the network data, executing network data management operation associated with the access request of the network data; the network data management operation associated with the access request of the network data comprises a file/directory operation of the network data; the file/directory operation of the network data comprises a file/directory read operation of metadata in the network data and a file/directory modification operation of the metadata in the network data.
2. The method for managing network data according to claim 1, wherein: the network data comprises metadata and file data; the request for access to the network data comprises: a read request of file data in the network data, a write request of file data in the network data, and/or an operation request of metadata in the network data.
3. The method for managing network data according to claim 2, wherein:
the distributed object file system cluster comprises a metadata storage module for storing metadata in network data and an object storage module for storing file data in the network data;
each network attached storage gateway comprises a metadata cache unit for caching metadata in network data, a file data cache unit for caching file data in the network data, and a high-availability management unit which is bound with different virtual IP addresses and detects heartbeat of each network attached storage gateway in the group of network attached storage gateways.
4. A method for managing network data according to claim 3, characterized in that: the network data reading operation associated with the reading request of the file data in the network data comprises the following steps:
after receiving the reading request of the network data, judging whether the network attached storage gateway caches the file data in the network data to be read after sending the reading request;
if the file data is cached, reading the file data from a file data caching unit of the network attached storage gateway, and feeding the read file data back to the client;
if the network data is not cached, the reading request of the network data is forwarded to the distributed object file system cluster, the network data is obtained and downloaded through random access, the downloaded network data is cached, and the file data of the network data is fed back to the client.
5. A method for managing network data according to claim 3, characterized in that: the network data writing operation associated with the writing request of the file data in the network data comprises the following steps:
after receiving the writing request of the network data, opening a file data cache unit of the network attached storage gateway, and randomly writing the file data of the network data into the file data cache unit to cache the written file data until the network data is completely written;
after the network data are completely written, closing a file data cache unit of the network attached storage gateway, and uploading the written file data to the distributed object file system; and simultaneously feeding back the information that the file data caching unit is closed to the client.
6. A method for managing network data according to claim 3, characterized in that: the file/directory operation of the metadata in the network data associated with the operation request of the metadata in the network data comprises:
after receiving a file/directory operation request of metadata in the network data, executing file/directory operation, and feeding back an operation result of the file/directory operation to the client; wherein the file/directory operation request of the metadata in the network data comprises a file/directory read operation request of the metadata in the network data and a file/directory update request of the metadata in the network data; the file/directory operation of the network data comprises a file/directory read operation of metadata in the network data and a file/directory modification operation of the metadata in the network data.
7. The method for managing network data according to claim 6, wherein: the file/directory reading operation of the network data comprises the following steps:
after receiving a file/directory reading operation request of metadata in the network data, directly reading the metadata from a metadata cache unit of the network attached storage gateway, and feeding back the read metadata to a client;
and periodically reading the metadata which is not cached in the metadata caching unit from the metadata storage module of the distributed object file system so as to update the cached metadata.
8. The method for managing network data according to claim 6, wherein: the file/directory modification operation of the network data comprises the following steps:
when the metadata in the metadata cache unit of the network attached storage gateway is updated, the updated metadata is written into the metadata storage module of the distributed object file system, and the metadata in the metadata cache unit of the network attached storage gateway is fed back to the client side as an updated operation result.
9. A network attached storage gateway is characterized in that the network attached storage gateway is applied to a network data storage service system comprising a plurality of groups of clients, a plurality of network attached storage gateway groups and a distributed object file system cluster; each group of network attached storage gateway groups comprises a plurality of network attached storage gateways, each group of clients is connected with one network attached storage gateway group, and the plurality of network attached storage gateway groups are connected with the distributed object file system cluster; the distributed object file system cluster is used for mapping the stored network data to the network attached storage gateways in the network attached storage gateway group so as to cache the network data; each network attached storage gateway comprises a metadata cache unit for caching metadata in network data, a file data cache unit for caching file data in the network data, and a high-availability management unit which is bound with different virtual IP addresses and detects heartbeat of each network attached storage gateway in the group of network attached storage gateways; the high-availability management unit detects heartbeat signals between the two gateways, when one network attached storage gateway is offline, the other network attached storage gateway cannot receive the heartbeat signals of the other network attached storage gateway, and the other network attached storage gateway takes over the VIP of the network attached storage gateway to continuously provide file storage service for the client; the network attached storage gateway comprises:
the metadata caching unit is used for caching metadata in the network data;
the object cache unit is used for caching the file data in the network data;
the operation unit is used for receiving an access request of the network data sent by the client side and bound with the operation unit; according to the access request of the network data, executing network data management operation associated with the access request of the network data; the network data management operation associated with the access request of the network data comprises a file/directory operation of the network data; the file/directory operation of the network data by the operation unit comprises a file/directory read operation of metadata in the network data and a file/directory modification operation of the metadata in the network data.
10. A storage service system for network data, comprising a plurality of groups of clients, a plurality of network attached storage gateway groups according to claim 9, and a distributed object file system cluster; each group of network attached storage gateway groups comprises a plurality of network attached storage gateways, each group of clients is connected with one network attached storage gateway group, and the plurality of network attached storage gateway groups are connected with the distributed object file system cluster; the distributed object file system cluster is used for mapping the stored network data to the network attached storage gateways in the network attached storage gateway group so as to cache the network data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611103290.8A CN106790434B (en) | 2016-12-05 | 2016-12-05 | Network data management method, network attached storage gateway and storage service system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611103290.8A CN106790434B (en) | 2016-12-05 | 2016-12-05 | Network data management method, network attached storage gateway and storage service system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106790434A CN106790434A (en) | 2017-05-31 |
| CN106790434B true CN106790434B (en) | 2020-03-27 |
Family
ID=58883374
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611103290.8A Active CN106790434B (en) | 2016-12-05 | 2016-12-05 | Network data management method, network attached storage gateway and storage service system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106790434B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107562889B (en) * | 2017-09-05 | 2022-02-18 | 郑州云海信息技术有限公司 | Metadata aging method and device |
| CN109783438B (en) * | 2018-12-05 | 2021-07-27 | 南京华讯方舟通信设备有限公司 | Distributed NFS system based on librados and construction method thereof |
| CN110109613A (en) * | 2019-03-14 | 2019-08-09 | 南京时沃信息科技有限公司 | A kind of object storage system based on disk |
| CN111212111B (en) * | 2019-12-17 | 2022-05-13 | 腾讯科技（深圳）有限公司 | Object storage service management method and electronic equipment |
| CN111339059A (en) * | 2020-03-25 | 2020-06-26 | 星辰天合（北京）数据科技有限公司 | NAS storage system based on distributed storage system Ceph |
| CN111581016B (en) * | 2020-04-14 | 2021-05-18 | 上海爱数信息技术股份有限公司 | Copy data management system and method for modern application |
| CN111581175B (en) * | 2020-04-29 | 2023-11-17 | 上海爱数信息技术股份有限公司 | File storage gateway system and data migration method thereof |
| CN111884917A (en) * | 2020-07-28 | 2020-11-03 | 浪潮云信息技术股份公司 | Gateway system based on micro service and gateway dynamic routing method |
| CN114079659A (en) * | 2020-08-13 | 2022-02-22 | 支付宝(杭州)信息技术有限公司 | Server of distributed storage system, data storage method and data access system |
| CN111953799A (en) * | 2020-08-25 | 2020-11-17 | 山东合天智汇信息技术有限公司 | Distributed management method and system for electronic license based on micro-service |
| CN113127420B (en) * | 2021-03-30 | 2023-03-14 | 山东英信计算机技术有限公司 | Metadata request processing method, device, equipment and medium |
| CN114265814B (en) * | 2022-03-01 | 2022-06-07 | 天津安锐捷技术有限公司 | Data lake file system based on object storage |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6697846B1 (en) * | 1998-03-20 | 2004-02-24 | Dataplow, Inc. | Shared file system |
| WO2005029251A2 (en) * | 2003-09-15 | 2005-03-31 | Neopath Networks, Inc. | Enabling proxy services using referral mechanisms |
| CN101764848A (en) * | 2010-01-12 | 2010-06-30 | 浪潮(北京)电子信息产业有限公司 | Method and device for transmitting network files |
| CN103607462A (en) * | 2013-11-22 | 2014-02-26 | 曙光信息产业股份有限公司 | Cluster NAS deployment system and deployment method thereof |
| CN105407044A (en) * | 2015-12-11 | 2016-03-16 | 西安交通大学 | Method for implementing cloud storage gateway system based on network file system (NFS) |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102192503B1 (en) * | 2015-04-01 | 2020-12-17 | 한국전자통신연구원 | Method and system for providing virtual desktop service using cache server |
-
2016
- 2016-12-05 CN CN201611103290.8A patent/CN106790434B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6697846B1 (en) * | 1998-03-20 | 2004-02-24 | Dataplow, Inc. | Shared file system |
| WO2005029251A2 (en) * | 2003-09-15 | 2005-03-31 | Neopath Networks, Inc. | Enabling proxy services using referral mechanisms |
| CN101764848A (en) * | 2010-01-12 | 2010-06-30 | 浪潮(北京)电子信息产业有限公司 | Method and device for transmitting network files |
| CN103607462A (en) * | 2013-11-22 | 2014-02-26 | 曙光信息产业股份有限公司 | Cluster NAS deployment system and deployment method thereof |
| CN105407044A (en) * | 2015-12-11 | 2016-03-16 | 西安交通大学 | Method for implementing cloud storage gateway system based on network file system (NFS) |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106790434A (en) | 2017-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106790434B (en) | Network data management method, network attached storage gateway and storage service system | |
| US8516159B2 (en) | Asynchronous file operations in a scalable multi-node file system cache for a remote cluster file system | |
| US9183213B2 (en) | Indirection objects in a cloud storage system | |
| US20040030731A1 (en) | System and method for accessing files in a network | |
| US8990257B2 (en) | Method for handling large object files in an object storage system | |
| US8788628B1 (en) | Pre-fetching data for a distributed filesystem | |
| JP4547263B2 (en) | Apparatus and method for processing data in a network | |
| US8055615B2 (en) | Method for efficient storage node replacement | |
| US9547706B2 (en) | Using colocation hints to facilitate accessing a distributed data storage system | |
| US20210344772A1 (en) | Distributed database systems including callback techniques for cache of same | |
| US20120317155A1 (en) | Hybrid Data Management System and Method for Managing Large, Varying Datasets | |
| US20110055494A1 (en) | Method for distributed direct object access storage | |
| US10783121B2 (en) | Techniques for optimizing data flows in hybrid cloud storage systems | |
| US20110307533A1 (en) | Data managing system, data managing method, and computer-readable, non-transitory medium storing a data managing program | |
| CN113377868B (en) | Offline storage system based on distributed KV database | |
| US20180034892A1 (en) | Multi-part upload | |
| JP2012141738A (en) | Information management method and information providing computer | |
| US10579597B1 (en) | Data-tiering service with multiple cold tier quality of service levels | |
| CN109144413A (en) | A kind of metadata management method and device | |
| US8612495B2 (en) | Computer and data management method by the computer | |
| CN104601724A (en) | Method and system for uploading and downloading file | |
| CN103473337A (en) | Massive catalogs and files oriented processing method in distributed type storage system | |
| CN112334891A (en) | Centralized storage for search servers | |
| US10055139B1 (en) | Optimized layout in a two tier storage | |
| CN115328857A (en) | File access method, device, client and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |