CN110781152B - Multi-protocol data sharing architecture - Google Patents
Multi-protocol data sharing architecture Download PDFInfo
- Publication number
- CN110781152B CN110781152B CN201910954198.XA CN201910954198A CN110781152B CN 110781152 B CN110781152 B CN 110781152B CN 201910954198 A CN201910954198 A CN 201910954198A CN 110781152 B CN110781152 B CN 110781152B
- Authority
- CN
- China
- Prior art keywords
- file
- data
- storage
- pool
- metadata
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000006243 chemical reaction Methods 0.000 claims abstract description 22
- 238000013507 mapping Methods 0.000 claims description 11
- 238000000034 method Methods 0.000 abstract description 9
- 238000013500 data storage Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000008520 organization Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 4
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/176—Support for shared access to files; File sharing support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/113—Details of archiving
Abstract
The application discloses a multi-protocol data sharing architecture, which comprises a file storage application, an object storage application, a file storage metadata pool, an object storage metadata pool and a file storage data pool; wherein: the file storage data pool is used for storing shared data in a file protocol mode; the file storage metadata pool is used for storing metadata corresponding to the shared data in a file protocol mode; the object storage metadata pool is used for storing metadata corresponding to the shared data in an object protocol mode; the file storage application is used for accessing a file storage metadata pool and a file storage data pool by adopting a file protocol; the object storage application is used for accessing the object storage metadata pool by adopting an object protocol and accessing the file storage data pool by utilizing a file protocol access interface obtained by self interface conversion. According to the method and the device, data sharing of the object protocol and the file protocol can be realized only by interface conversion, and the problems of efficiency and occupied storage space existing in data format conversion are effectively avoided.
Description
Technical Field
The application relates to the technical field of cloud computing and big data scene data centers, in particular to a multi-protocol data sharing architecture.
Background
In cloud computing and big data era, mass data storage is generally realized by adopting a plurality of protocols, such as a file protocol and an object protocol, and data storage realized by adopting the file protocol and the object protocol is respectively file protocol storage and object protocol storage; but the formats of data stored by different protocols cannot be shared by multiple protocols due to different organization forms of corresponding metadata. In this case, if an application needs to access by using both a file protocol and an object protocol, it is a conventional practice to copy data stored in the file protocol into an object protocol store, so as to convert the data into a format and make the object protocol accessible, and vice versa.
Content of application
The application aims to provide a multi-protocol data sharing architecture, which can solve the problems of efficiency of format conversion and storage space occupation existing in the prior art when multi-protocol data sharing is realized.
In order to achieve the above purpose, the present application provides the following technical solutions:
a multi-protocol data sharing architecture comprises a file storage application, an object storage application, a file storage metadata pool, an object storage metadata pool and a file storage data pool; wherein:
the file storage data pool is used for storing shared data in a file protocol mode;
the file storage metadata pool is used for storing metadata corresponding to the shared data in a file protocol mode;
the object storage metadata pool is used for storing metadata corresponding to the shared data in an object protocol mode;
the file storage application is used for accessing the file storage metadata pool and the file storage data pool by adopting a file protocol;
and the object storage application is used for accessing the object storage metadata pool by adopting an object protocol and accessing the file storage data pool by utilizing a file protocol access interface obtained by self interface conversion.
Preferably, the system further comprises a shared directory and a file storage directory, wherein:
the file storage data pool is also used for storing file storage data in a file protocol mode;
the file storage metadata pool is also used for storing metadata corresponding to the file storage data in a file protocol mode;
the file storage data is data under the file storage directory, and correspondingly, the file storage application is also used for realizing the access of the file storage data by accessing the file storage directory by adopting a file protocol;
the shared data is data under the shared directory, and correspondingly, the file storage application and the object storage application are used for realizing the access of the shared data by accessing the shared directory.
Preferably, the system further comprises an object storage data pool, wherein:
the object storage data pool is used for storing object storage data in an object protocol mode;
the object storage metadata pool is also used for storing metadata corresponding to the object storage data in an object protocol mode;
the object storage application is further configured to access the object storage data pool using an object protocol.
Preferably, the system further comprises a file storage update thread, an object storage update thread, a file storage asynchronous thread and an object storage asynchronous thread, wherein:
the file storage updating thread is used for updating corresponding metadata to the file storage metadata pool after the file storage application changes the shared data;
the object storage updating thread is used for updating corresponding metadata to the object storage metadata pool after the object storage application changes the shared data;
the file storage asynchronous thread is used for asynchronously updating corresponding metadata to the object storage metadata pool after the metadata corresponding to the shared data in the file system metadata pool is changed;
and the object storage asynchronous thread is used for asynchronously updating the corresponding metadata to the file storage metadata pool after the metadata corresponding to the shared data in the object system metadata pool is changed.
Preferably, the system further comprises a file storage naming thread and an object storage naming thread, wherein:
the file storage naming thread is used for naming the created file by using a naming rule of a file protocol when the file storage application creates the file, and writing the mapping relation between the created file and the corresponding name into the file storage metadata pool as corresponding metadata;
and the object storage naming thread is used for naming the created file by using a naming rule of an object protocol when the object storage application creates the file, and writing the mapping relation between the created file and the corresponding name into the object storage metadata pool as corresponding metadata.
Preferably, the method also comprises the steps of updating the thread in real time;
the real-time update thread is configured to update the corresponding metadata to the file storage metadata pool and the object storage metadata pool at the same time after the file storage application or the object storage application changes the shared data.
Preferably, the data written by the file storage application in the shared directory all belong to a designated user, the data written by the object storage application in the shared directory belong to a corresponding object user, the data of the designated user is visible to the object user, and the data of the object user is visible to the designated user.
Preferably, the shared directory includes a first-layer directory and a second-layer directory under the first-layer directory, where the first-layer directory corresponds to a bucket in the object protocol, and the second-layer directory corresponds to an object under the bucket.
Preferably, the method also comprises big data application;
and the big data application is used for accessing the file storage metadata pool and the file storage data pool by using a file protocol access interface obtained by self interface conversion.
In the multi-protocol data sharing architecture disclosed by the application, shared data needing to be shared by multiple protocols are stored in a file storage data pool in a file protocol mode, corresponding metadata are stored in a file storage metadata pool in a file protocol mode and stored in an object storage metadata pool in an object protocol mode, so that file storage application and object storage application can be positioned to the shared data by accessing the corresponding metadata pool, further the file storage application can directly access the shared data by adopting the file protocol, and the object storage application can convert an interface of the object storage application into a file protocol access interface to access the shared data; therefore, based on the framework disclosed by the application, the data sharing of the object protocol and the file protocol can be realized only by the action of realizing interface conversion by the object storage application, the format conversion of the data is not needed, and the problems of efficiency and storage space occupation existing in the data format conversion in the prior art are correspondingly solved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a first schematic diagram of a multi-protocol data sharing architecture according to an embodiment of the present application;
fig. 2 is a second schematic diagram of a multi-protocol data sharing architecture according to an embodiment of the present application;
fig. 3 is a schematic diagram of a process of implementing asynchronous metadata update by using a corresponding thread in a multi-protocol data sharing architecture according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, a schematic diagram of a multi-protocol data sharing architecture provided in an embodiment of the present application is shown, where the multi-protocol data sharing architecture may include a file storage application, an object storage application, a file storage metadata pool, an object storage metadata pool, and a file storage data pool; wherein:
the file storage data pool is used for storing shared data in a file protocol mode;
the file storage metadata pool is used for storing metadata corresponding to the shared data in a file protocol mode;
the object storage metadata pool is used for storing metadata corresponding to the shared data in an object protocol mode;
the file storage application is used for accessing a file storage metadata pool and a file storage data pool by adopting a file protocol;
the object storage application is used for accessing the object storage metadata pool by adopting an object protocol and accessing the file storage data pool by utilizing a file protocol access interface obtained by self interface conversion.
The file protocol and the object protocol are both protocols (storage protocols for short) used for realizing data storage, and have the same meaning as corresponding concepts in the prior art; the multi-protocol data sharing architecture is an architecture for realizing data sharing among different storage protocols, wherein shared data is data which needs to be accessed by a file storage application and an object storage application at the same time, namely data which needs to be shared, and metadata is information data which can be positioned at the position of the shared data; the file storage data pool is used as bottom storage and is used for storing shared data in a file protocol mode; the file storage metadata pool and the object storage metadata pool are respectively used for storing metadata corresponding to the shared data in a file protocol mode and an object protocol mode; the file storage application is an application needing to access shared data by adopting a file protocol, and the object storage application is an application needing to access the shared data by adopting an object protocol.
The process of implementing access to shared data based on a multi-protocol data sharing architecture may include: when the file storage application needs to access shared data, if new shared data needs to be written, the new shared data can be written into the file storage data pool, then the written new shared data corresponding metadata is updated to the file storage metadata pool and the object storage metadata pool, if the shared data needs to be read, the position of the shared data needing to be read can be determined in the file storage metadata pool, then the shared data is read at the corresponding position in the file storage data pool, if the shared data needs to be modified, the position of the shared data needing to be modified can be determined in the file storage metadata pool, then the shared data at the corresponding position in the file storage data pool can be modified, if the modification of the shared data causes the change of the corresponding metadata, the new metadata corresponding to the modified shared data needs to be updated to the file storage metadata pool and the object storage metadata pool; the object storage application can access the object storage metadata pool by adopting an object protocol, and simultaneously can access the file storage metadata pool and the file storage data pool by adopting a file protocol through converting a self interface into a file protocol access interface, when the object storage application needs to access shared data, if new shared data needs to be written, the position where the new shared data needs to be written can be selected from the object storage metadata pool, then the file protocol access interface obtained through conversion accesses the file storage metadata pool by adopting the file protocol, the position where the new shared data needs to be written is determined, the new shared data is written into the corresponding position of the file storage data pool (or an object protocol storage data organization structure is changed into a file protocol storage data organization structure), then updating the written new shared data corresponding metadata to a file storage metadata pool and an object storage metadata pool, if the shared data needs to be read, selecting the position of the shared data needing to be read from the object storage metadata pool, accessing the file storage metadata pool by a file protocol through a file protocol access interface obtained by conversion, determining the position of the shared data needing to be read, then reading the shared data at the corresponding position in the file storage metadata pool, if the shared data needs to be modified, selecting the position of the shared data needing to be modified from the object storage metadata pool, accessing the file storage metadata pool by a file protocol through a file protocol access interface obtained by conversion, determining the position of the shared data needing to be modified, and then modifying the shared data at the corresponding position in the file storage metadata pool, if the modification of the shared data causes the corresponding metadata to change, the new metadata corresponding to the modified shared data needs to be updated to the file storage metadata pool and the object storage metadata pool.
It should be noted that, in the multi-protocol data sharing architecture of the present application, the file protocol storage and the object protocol storage may have corresponding thread pools, and each thread that implements the corresponding file protocol storage or object protocol storage is included in each thread pool, and specifically, when the file storage application or the object storage application needs to implement shared data access, the thread pool of the file storage application or the object storage application responds to the requirement of the corresponding application, and implements corresponding functions, such as implementing metadata update, reading data, writing data, and modifying data, by invoking the corresponding thread.
In the multi-protocol data sharing architecture disclosed by the application, shared data needing to be shared by multiple protocols are stored in a file storage data pool in a file protocol mode, corresponding metadata are stored in a file storage metadata pool in a file protocol mode and stored in an object storage metadata pool in an object protocol mode, so that file storage application and object storage application can be positioned to the shared data by accessing the corresponding metadata pool, further the file storage application can directly access the shared data by adopting the file protocol, and the object storage application can convert an interface of the object storage application into a file protocol access interface to access the shared data; therefore, based on the framework disclosed by the application, the data sharing of the object protocol and the file protocol can be realized only by the action of realizing interface conversion by the object storage application, the format conversion of the data is not needed, and the problems of efficiency and storage space occupation existing in the data format conversion in the prior art are correspondingly solved.
The multi-protocol data sharing architecture provided by the embodiment of the application may further include a sharing directory and a file storage directory, wherein:
the file storage data pool is also used for storing file storage data in a file protocol mode;
the file storage metadata pool is also used for storing metadata corresponding to the file storage data in a file protocol mode;
the file storage data is data under a file storage directory, and correspondingly, the file storage application is also used for realizing the access of the file storage data by accessing the file storage directory by adopting a file protocol;
the shared data is data under a shared directory, and correspondingly, the file storage application and the object storage application are used for realizing the access of the shared data by accessing the shared directory.
The embodiment is provided with a sharing directory and a file storage directory, wherein the sharing directory corresponds to sharing data needing to realize multi-protocol data sharing, and the file storage directory corresponds to file storage data which can only be accessed by adopting a file protocol; correspondingly, the file storage application can respectively access the shared data in the shared directory and the file storage data in the file storage directory by adopting a file protocol. Specifically, when the file storage application needs to access data in the file storage data pool, a directory (shared directory or file storage directory) that needs to be accessed may be selected based on the access file storage metadata pool, and then the file storage metadata pool and the file storage data pool are accessed under the directory to implement access to corresponding data (specifically, implementing access to corresponding data includes the processes of writing data, reading data, and modifying data described in the above embodiments). As can be seen, in this embodiment, the file storage application has two paths for accessing the data stored in the file storage data pool, one is access implemented through a shared directory, and the other is access implemented through a file storage directory, so that when sharing of partial data in the file storage data pool is implemented, partial data that can only be accessed by using a file protocol is retained, and when a sharing requirement is met, protection of data that is not allowed to be shared with the object protocol storage by the file protocol storage is also met.
The multi-protocol data sharing architecture provided by the embodiment of the present application may further include an object storage data pool, where:
the object storage data pool is used for storing object storage data in an object protocol mode;
the object storage metadata pool is also used for storing metadata corresponding to the object storage data in an object protocol mode;
the object store application is further configured to access the object store data pool using an object protocol.
The object storage data is data to be accessed by adopting an object protocol, and an object storage data pool is arranged in the embodiment and is used for storing the object storage data in an object protocol mode so as to enable the object storage application to realize the access of the object storage data by adopting the object protocol; specifically, when the object storage application needs to access data, the location of the data to be accessed may be selected based on the object storage metadata pool, if the selected location is in the shared directory, the shared directory is accessed through the converted file protocol access interface, and then access to the shared data in the shared directory is implemented (the specific access to the shared data includes the processes of writing data, reading data, and modifying data described in the above embodiments), and if the selected location is in the object storage data pool, the data access may be directly implemented at the corresponding location in the object storage data pool, so that the object storage application may implement access to not only the shared data stored in the file protocol manner through the conversion interface, but also the object storage data stored in the object protocol manner through the object storage, the architecture function is more comprehensive; in addition, data which does not need to be shared is directly stored as object storage data, interface conversion is not needed when access is achieved, and access speed of corresponding data is increased.
The structure schematic diagram of the multi-protocol data sharing architecture disclosed in the present application when having a shared directory, a file storage directory, and an object storage data pool may be as shown in fig. 2, wherein a specific directory stored by a file protocol is used as the shared directory capable of implementing multi-protocol data sharing, and an access operation for the shared directory refreshes the file storage metadata pool and the object storage metadata pool, respectively, so as to implement multi-protocol sharing of the specific directory stored by the file protocol. And the file protocol storage and the object protocol storage respectively have respective storage pools for data organization and metadata organization of the metadata pools, namely paths for storing data in respective data formats, so that simultaneous deployment of the file protocol storage and the object protocol storage is realized, and respective characteristics are reserved. It can be seen that, the present application is based on file protocol storage, and implements a data sharing mechanism of file protocol and object protocol, under different application scenarios, a copy of data stored at the bottom layer of the file protocol storage can be accessed by the file protocol and the object protocol at the same time, and the file protocol and the object protocol can access respective stored data at the bottom layer, specifically, for part of data that needs to implement multi-protocol data sharing, put into the sharing directory, provide upward mutual access while storing the file protocol and the object protocol, for the data that does not need to implement multi-protocol data sharing, put into the file storage directory and the object storage data pool, respectively, thereby achieving the best combination of efficiency and storage utilization, reducing data format conversion, and enhancing access efficiency, and simultaneously enhancing storage utilization due to reducing the storage of the same data and different protocols, the data access method has the advantages that different access interface requirements are met, different protocols are adopted for data access, efficient and convenient access modes and good user experience are achieved, and competitiveness of the data center is improved.
The multi-protocol data sharing architecture provided by the embodiment of the application may further include a file storage updating thread, an object storage updating thread, a file storage asynchronous thread, and an object storage asynchronous thread, wherein:
the file storage updating thread is used for updating corresponding metadata to a file storage metadata pool after the file storage application changes the shared data;
the object storage updating thread is used for updating corresponding metadata to the object storage metadata pool after the object storage application changes the shared data;
the file storage asynchronous thread is used for asynchronously updating corresponding metadata to the object storage metadata pool after the metadata corresponding to the shared data in the file system metadata pool is changed;
and the object storage asynchronous thread is used for asynchronously updating the corresponding metadata to the file storage metadata pool after the metadata corresponding to the shared data in the object system metadata pool is changed.
The file storage updating thread and the file storage asynchronous thread can belong to a data pool stored by a file protocol, and the object storage updating thread and the object storage asynchronous thread can belong to a data pool stored by an object protocol. It should be noted that the sharing of the metadata may be non-real-time, specifically, after the file storage application changes the shared data, the file storage update thread may update the corresponding metadata to the file storage metadata pool, and then the file storage asynchronous thread asynchronously updates (or refreshes) the metadata corresponding to the object storage metadata pool; after the object storage application changes the shared data, the object storage updating thread can update the corresponding metadata to the object storage metadata pool, and then the object storage asynchronous thread asynchronously updates the corresponding metadata in the file storage metadata pool; therefore, non-real-time sharing of metadata corresponding to shared data is achieved through the asynchronous metadata updating mode, the characteristics that the system performance can be improved and the system robustness can be improved through the asynchronous updating mode are fully utilized, and the performance and the robustness can be improved when the multi-protocol data sharing architecture achieves updating of the metadata corresponding to the shared data.
The multi-protocol data sharing architecture provided by the embodiment of the application may further include a file storage naming thread and an object storage naming thread, wherein:
the file storage naming thread is used for naming the created file by using a naming rule of a file protocol when the file storage application creates the file, and writing the mapping relation between the created file and the corresponding name into a file storage metadata pool as corresponding metadata;
the object storage naming thread is used for naming the created file by using a naming rule of an object protocol when the object storage application creates the file, and writing the mapping relation between the created file and the corresponding name into the object storage metadata pool as corresponding metadata.
The file store naming thread may belong to a thread pool of a file protocol store and the object store naming thread may belong to a thread pool of an object protocol store.
It should be noted that, when a file storage application needs to create a file to implement data storage therein, an oid (identifier) rule stored at a bottom layer is implemented according to a slice rule and a naming rule stored in a file protocol, in this embodiment, a part related to the naming rule is mainly described, specifically, a file storage naming thread names the created file according to the naming rule of the file protocol, and stores a mapping relationship between the file (file) and a name (ids) of the file into a file storage metadata pool, and then a file storage asynchronous thread calls a synchronous metadata interface provided by an object storage metadata pool to asynchronously refresh the mapping relationship to the object storage metadata pool, so that the object storage application can access the object storage application to know the mapping relationship to implement location selection and determination; correspondingly, when an object storage application needs to create a file to implement data storage therein, oid rules of the underlying storage are implemented according to the slicing rules and the naming rules stored in the object protocol. The naming of the corresponding file and the determination of the metadata are realized through the method, so that the applications of different storage protocols can be positioned to the corresponding file through the mapping relation, and the access to the file is convenient to realize.
In addition, after the object storage naming thread stores the obtained mapping relation into the object storage metadata pool, the file storage metadata pool can also establish a link between the files created by the object storage application in a soft link mode, and further can directly position the corresponding file based on the link, and the mode is simple and easy to implement and has higher file positioning speed; of course, other settings can be performed according to actual needs, and are within the protection scope of the present application.
The multi-protocol data sharing architecture provided by the embodiment of the application can further comprise a real-time updating thread;
the real-time updating thread is used for updating corresponding metadata to the file storage metadata pool and the object storage metadata pool at the same time after the file storage application or the object storage application changes the shared data.
In this embodiment, after the file storage application or the object storage application changes the shared data, the real-time update thread updates the corresponding metadata to the file storage metadata pool and the object storage metadata pool at the same time; specifically, the real-time update thread may belong to a thread pool stored in a file protocol, and correspondingly, after the shared data is changed, the real-time update thread may directly update corresponding metadata in a metadata pool of the file storage, and simultaneously update the metadata corresponding to the changed shared data into the metadata pool of the object storage by calling a metadata synchronization interface provided by the metadata pool of the object storage; by the mode of refreshing the metadata at the same time, the updating of the metadata in the object storage metadata pool and the file storage metadata pool can be quickly realized, and the real-time performance and the realization efficiency of the multi-protocol data sharing architecture in realizing the updating of the metadata corresponding to the shared data are improved.
According to the multi-protocol data sharing architecture provided by the embodiment of the application, data written by the file storage application in the shared directory belong to the designated users, data written by the object storage application in the shared directory belong to the corresponding object users, the data of the designated users are visible to the object users, and the data of the object users are visible to the designated users.
Since the units of the object protocol storage are sequentially users, buckets, objects, and the like, in order to ensure that the shared data is visible for both the file protocol application and the object protocol application, in this embodiment, it is set that the data written by the file storage application under the shared directory all belong to the designated users (a preset default designated user s3root is a super user, and the data written by other users are visible for the super user), while the data written by the object storage application under the shared directory all belong to corresponding object users (a plurality of object users may be included, such as s3uer1, s3uer2, s3user3, and the like), so that the data of the object users are visible for the designated users, that is, all the data written by the designated user is visible for the designated user, and all the shared data written by the designated user is visible for the object user, so that the shared data can be visible for both the file storage application and the object protocol application, thereby enabling it to access the shared data.
According to the multi-protocol data sharing architecture provided by the embodiment of the application, the shared directory comprises a first-layer directory and a second-layer directory under the first-layer directory, wherein the first-layer directory corresponds to a bucket in an object protocol, and the second-layer directory corresponds to an object under the bucket.
In order to make the shared directory visible to the object storage application as a unit corresponding to the object protocol storage, in this embodiment, a corresponding relationship is set between the shared directory and the bucket, and a corresponding relationship is set between a two-layer directory (or a subdirectory) under the shared directory and the object, so that the shared directory is a first-layer directory for the file protocol application and is a bucket for the object storage application, the two-layer directory of the shared directory is a subdirectory under the first-layer directory for the file protocol application, and the object is for the object storage application, and if the first-layer directory of the shared directory is a/, b/, and c/three horizontal directories, the object storage application sees a, b, and c buckets; the directory of a/d/e is a directory with an object d/e in a barrel a for an object storage application; therefore, the object storage application can conveniently realize the access of the shared data based on the shared directory.
The multi-protocol data sharing architecture provided by the embodiment of the application can further comprise big data application;
and the big data application is used for accessing the file storage metadata pool and the file storage data pool by using the file protocol access interface obtained by self interface conversion.
In the embodiment, big data application can be added, the big data application realizes data storage in a way of HDFS protocol, in order to enable the big data application to realize data sharing, an interface of the big data application can be converted to obtain a file protocol access interface, and then the big data application can realize access to a file storage metadata pool and a file storage data pool in a way of file protocol based on the file protocol access interface, so that data sharing of more protocols is further realized; in addition, as the big data application is an upper application, the big data application can realize data access based on a file protocol with the same function as the file storage application by using a file protocol access interface, namely the big data application can access shared data and also can access file storage data, thereby fully realizing the data sharing between the file protocol and the HDFS protocol; and when the big data application realizes the access of shared data based on the shared directory, the data sharing among three storage protocols, namely an HDFS protocol, a file protocol and an object protocol, can be realized. In addition, when there is a big data application, a schematic diagram of implementing asynchronous updating metadata by using a multi-protocol data sharing architecture of the present application may be as shown in fig. 3, where a file storage module represents a module implementing file protocol storage, and includes a thread pool of file protocol storage, an object storage module represents a module implementing object protocol storage, and includes a thread pool of object protocol storage, different applications implement corresponding functions by calling threads in corresponding thread pools, dashed lines with arrows represent asynchronous updating metadata, and solid lines with arrows represent access to corresponding metadata pools and data pools by calling threads.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (8)
1. A multi-protocol data sharing architecture is characterized by comprising a file storage application, an object storage application, a file storage metadata pool, an object storage metadata pool and a file storage data pool; wherein:
the file storage data pool is used for storing shared data in a file protocol mode;
the file storage metadata pool is used for storing metadata corresponding to the shared data in a file protocol mode;
the object storage metadata pool is used for storing metadata corresponding to the shared data in an object protocol mode;
the file storage application is used for accessing the file storage metadata pool and the file storage data pool by adopting a file protocol;
the object storage application is used for accessing the object storage metadata pool by adopting an object protocol and accessing the file storage data pool by utilizing a file protocol access interface obtained by self interface conversion;
the multi-protocol data sharing architecture further comprises a file storage updating thread, an object storage updating thread, a file storage asynchronous thread and an object storage asynchronous thread, wherein:
the file storage updating thread is used for updating corresponding metadata to the file storage metadata pool after the file storage application changes the shared data;
the object storage updating thread is used for updating corresponding metadata to the object storage metadata pool after the object storage application changes the shared data;
the file storage asynchronous thread is used for asynchronously updating corresponding metadata to the object storage metadata pool after the metadata corresponding to the shared data in the file storage metadata pool is changed;
and the object storage asynchronous thread is used for asynchronously updating the corresponding metadata to the file storage metadata pool after the metadata corresponding to the shared data in the object storage metadata pool is changed.
2. The architecture of claim 1, further comprising a shared directory and a file storage directory, wherein:
the file storage data pool is also used for storing file storage data in a file protocol mode;
the file storage metadata pool is also used for storing metadata corresponding to the file storage data in a file protocol mode;
the file storage data is data under the file storage directory, and correspondingly, the file storage application is also used for realizing the access of the file storage data by accessing the file storage directory by adopting a file protocol;
the shared data is data under the shared directory, and correspondingly, the file storage application and the object storage application are used for realizing the access of the shared data by accessing the shared directory.
3. The architecture of claim 2, further comprising an object store data pool, wherein:
the object storage data pool is used for storing object storage data in an object protocol mode;
the object storage metadata pool is also used for storing metadata corresponding to the object storage data in an object protocol mode;
the object storage application is further configured to access the object storage data pool using an object protocol.
4. The architecture of claim 3, further comprising a file store naming thread and an object store naming thread, wherein:
the file storage naming thread is used for naming the created file by using a naming rule of a file protocol when the file storage application creates the file, and writing the mapping relation between the created file and the corresponding name into the file storage metadata pool as corresponding metadata;
and the object storage naming thread is used for naming the created file by using a naming rule of an object protocol when the object storage application creates the file, and writing the mapping relation between the created file and the corresponding name into the object storage metadata pool as corresponding metadata.
5. The architecture of claim 3, further comprising a real-time update thread;
the real-time update thread is configured to update the corresponding metadata to the file storage metadata pool and the object storage metadata pool at the same time after the file storage application or the object storage application changes the shared data.
6. The architecture of claim 4 or 5, wherein the data written by the file storage application under the shared directory all belong to a specified user, the data written by the object storage application under the shared directory belong to a corresponding object user, the data of the specified user is visible to the object user, and the data of the object user is visible to the specified user.
7. The architecture of claim 6, wherein the shared directory comprises a first-level directory and a second-level directory below the first-level directory, wherein the first-level directory corresponds to a bucket in the object protocol, and wherein the second-level directory corresponds to an object below the bucket.
8. The architecture of claim 7, further comprising a big data application;
and the big data application is used for accessing the file storage metadata pool and the file storage data pool by using a file protocol access interface obtained by self interface conversion.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910954198.XA CN110781152B (en) | 2019-10-09 | 2019-10-09 | Multi-protocol data sharing architecture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910954198.XA CN110781152B (en) | 2019-10-09 | 2019-10-09 | Multi-protocol data sharing architecture |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110781152A CN110781152A (en) | 2020-02-11 |
| CN110781152B true CN110781152B (en) | 2022-04-22 |
Family
ID=69384912
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910954198.XA Active CN110781152B (en) | 2019-10-09 | 2019-10-09 | Multi-protocol data sharing architecture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110781152B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220100878A1 (en) * | 2020-09-25 | 2022-03-31 | EMC IP Holding Company LLC | Facilitating an object protocol based access of data within a multiprotocol environment |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114116641A (en) * | 2020-08-31 | 2022-03-01 | 华为技术有限公司 | Method and device for synchronizing metadata |
| CN114466083B (en) * | 2022-01-19 | 2023-11-17 | 北京星辰天合科技股份有限公司 | Data storage system supporting protocol interworking |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5535375A (en) * | 1992-04-20 | 1996-07-09 | International Business Machines Corporation | File manager for files shared by heterogeneous clients |
| US7194550B1 (en) * | 2001-08-30 | 2007-03-20 | Sanera Systems, Inc. | Providing a single hop communication path between a storage device and a network switch |
| CN104901877A (en) * | 2015-06-17 | 2015-09-09 | 燕山大学 | Multi-interface self-adaptive wireless heterogeneous network protocol conversion method and communication device |
| CN109040144A (en) * | 2018-10-19 | 2018-12-18 | 京信通信系统（中国）有限公司 | Things-internet gateway data processing method, device, storage medium and gateway |
-
2019
- 2019-10-09 CN CN201910954198.XA patent/CN110781152B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5535375A (en) * | 1992-04-20 | 1996-07-09 | International Business Machines Corporation | File manager for files shared by heterogeneous clients |
| US7194550B1 (en) * | 2001-08-30 | 2007-03-20 | Sanera Systems, Inc. | Providing a single hop communication path between a storage device and a network switch |
| CN104901877A (en) * | 2015-06-17 | 2015-09-09 | 燕山大学 | Multi-interface self-adaptive wireless heterogeneous network protocol conversion method and communication device |
| CN109040144A (en) * | 2018-10-19 | 2018-12-18 | 京信通信系统（中国）有限公司 | Things-internet gateway data processing method, device, storage medium and gateway |
Non-Patent Citations (1)
| Title |
|---|
| 空管接口信息服务平台的设计与实现;韩剑锋等;《微型电脑应用》;20130720;第30卷(第07期);6-8 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220100878A1 (en) * | 2020-09-25 | 2022-03-31 | EMC IP Holding Company LLC | Facilitating an object protocol based access of data within a multiprotocol environment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110781152A (en) | 2020-02-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110781152B (en) | Multi-protocol data sharing architecture | |
| CN105933376B (en) | A kind of data manipulation method, server and storage system | |
| TW202021328A (en) | Unified resource scheduling coordinator and method thereof for creating a virtual machine and/or container, and unified resource scheduling system | |
| CN110377436A (en) | Data storage access method, equipment and the device of persistence memory | |
| CN110198231A (en) | Capacitor network management method and system and middleware for multi-tenant | |
| Adya et al. | Fast key-value stores: An idea whose time has come and gone | |
| WO2021129477A1 (en) | Data synchronization method and related device | |
| US11080207B2 (en) | Caching framework for big-data engines in the cloud | |
| US10102230B1 (en) | Rate-limiting secondary index creation for an online table | |
| CN113626550B (en) | Image tile map service method based on triple bidirectional index and optimized cache | |
| WO2014110940A1 (en) | A method, apparatus and system for storing, reading the directory index | |
| CN108089912A (en) | A kind of virtual machine surpasses emerging system construction method and device with container | |
| CN105740469B (en) | Storage server and metadata access method | |
| US10747739B1 (en) | Implicit checkpoint for generating a secondary index of a table | |
| US9875270B1 (en) | Locking item ranges for creating a secondary index from an online table | |
| US20240061712A1 (en) | Method, apparatus, and system for creating training task on ai training platform, and medium | |
| CN103390018A (en) | Web service data modeling and searching method based on SDD (service data description) | |
| CN107566474A (en) | The multi-source heterogeneous data of the three-dimensional earth based on mobile platform are downloaded and integrated approach | |
| CN106570113A (en) | Cloud storage method and system for mass vector slice data | |
| JP2001101042A (en) | System and method for managing data | |
| US11385900B2 (en) | Accessing queue data | |
| CN109614373A (en) | A kind of storage of small documents storage architecture and read method | |
| US9898614B1 (en) | Implicit prioritization to rate-limit secondary index creation for an online table | |
| WO2023179784A1 (en) | Data processing method and apparatus | |
| CN115361389B (en) | Cloud computing instance creating method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |