CN104113597A - Multi- data-centre hadoop distributed file system (HDFS) data read-write system and method - Google Patents
Multi- data-centre hadoop distributed file system (HDFS) data read-write system and method Download PDFInfo
- Publication number
- CN104113597A CN104113597A CN201410344218.9A CN201410344218A CN104113597A CN 104113597 A CN104113597 A CN 104113597A CN 201410344218 A CN201410344218 A CN 201410344218A CN 104113597 A CN104113597 A CN 104113597A
- Authority
- CN
- China
- Prior art keywords
- data
- metadata
- data center
- client
- hdfs
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a multi- data-centre hadoop distributed file system (HDFS) data read-write system and method. The system and the method are characterized in that a global metadata server is established, used for storing and managing global metadata information and responsible for receiving data read-write access requests of a client, and HDFS data centers are selected according to a preset scheduling algorithm, data read-write operations are interactively performed between the client and the selected data centers, and after the operations are finished, change information of metadata is synchronized to the global metadata server by metadata nodes of the data centers. By means of the multi- data-centre HDFS data read-write system and method, data read-write accesses of multiple HDFS data centers are achieved, a unified data access interface is provided, and resource and data sharing of the multiple HDFS data centers can be effectively achieved.
Description
Technical field
The present invention relates to computer cloud memory technology, particularly a kind of data read-write system and method based on HDFS distributed document system.
Background technology
Cloud storage is in cloud computing (Cloud Computing) conceptive extension and develops.Cloud storage refers to by functions such as cluster, grid or distributed file systems, a large amount of various dissimilar memory devices in network are gathered to collaborative work by application software, data storages and Operational Visit function are externally provided jointly, and guarantee the fail safe etc. of data.
The technology that the current Hadoop project distributed file system with the Apache of community that increases income (Hadoop Distributed File System is called for short HDFS) and multiple programming framework Hadoop MapReduce are representative becomes the mainstream technology of mass data storage and analyzing and processing gradually.Wherein, HDFS becomes one of most popular distributed file system gradually, is the main flow file system of current structure cloud storage.
HDFS system configuration as shown in Figure 1, is mainly comprised of metadata node NameNode, back end DataNode and client Client.Wherein, NameNode also claims Master node, is in charge of name space and the data block map information of HDFS, configuration replication policy, and process client-requested.DataNode, also referred to as Slave node, stores actual data, the read-write operation of executing data piece, and periodically the data block information of storage is reported to NameNode.Client Client, for cutting data file, accesses or manages HDFS by order line; Mutual with NameNode, obtain file location information; Mutual with DataNode, carry out data read and write operation.
At present, HDFS is widely used in data center by numerous enterprises, colleges and universities, scientific research institutions etc., becomes gradually the basic storage system of data center, is bearing mass data storage task.Along with more and more foundation that is dispersed in the middle-size and small-size data center of independence everywhere, how effectively shared the storage resources of each data center and data are, unified data access interface is provided how to outer layer service, becomes one of core difficult problem of restriction cloud storage system high speed development and extensive use.The current HDFS reading and writing data technology about many data centers, there are no concrete open report, is current technical problem urgently to be resolved hurrily.
Summary of the invention
The object of this invention is to provide a kind of read-write that can be data provides unified access interface, realizes most according to central information and effective shared HDFS data read-write system and the method for resource.
For reaching above object, invention takes following technical scheme to be achieved:
A HDFS data read-write system for many data centers, is characterized in that, comprises a global metadata server, a n data center, a client, and all there are a metadata node and a plurality of back end in each data center; Between global metadata server and client He Ge data center metadata node, adopt wide area network to link, between the metadata node of each data center and back end, by local area network (LAN), linked; Global metadata server, for the metadata information of the store and management overall situation, is responsible for each data center and distributes metadata NameSpace; The metadata node of each data center all comprises a GMS plugin module, is responsible for to global metadata server registration and regularly reports data center's resource using status and metadata information; Global metadata server is responsible for receiving client HDFS reading and writing data access request, and selects according to default dispatching algorithm the data center meeting the demands; The metadata node at client-access selected data center, by this metadata node, carried out the scheduling of HDFS reading and writing data, client is after HDFS reading and writing data completes, and the metadata node of data center is synchronized to global metadata server by the change information of metadata again.
A HDFS data read-write method for many data centers, is characterized in that, comprises the large step of read and write two:
The first step, HDFS data are read, and comprising:
(1) set up global metadata server, for the metadata information of the store and management overall situation; Global metadata server Wei Ge data center distributes NameSpace, and metadata information is reported to global metadata server by each data center;
(2) global metadata server receives client read data request, selects to meet the data center of reading requirement by preset algorithm, returns to the metadata node information at selected data center;
(3) metadata node of client-access data center, metadata node returns to client according to default dispatching algorithm data block and back end information;
(4) client and back end carry out alternately, and reading out data has read rear notice metadata node, and read data has operated;
Second step, HDFS data are write, and comprising:
(1) step of reading with HDFS data (1);
(2) global metadata server receives client read data request, by preset algorithm, selects to meet the data center that writes requirement, returns to the metadata node information at selected data center;
(3) metadata node of the selected HDFS of client-access data center, metadata node creates metadata information, and according to preset algorithm distribute data node, and back end information is returned to client;
(4) client and back end carry out alternately, carrying out data writing operation, have write rear notice metadata node; During client data writing, adopt piecemeal writing mechanism, data block copy copy is completed automatically by back end, and all data blocks all write successfully notifies metadata node to write afterwards;
(5), after ablation process completes, the metadata node of data center is synchronized to global metadata server by the change information of metadata.
In said method, described client read data request comprises any feature of file path, data block index, buffer size; Described client write data requests comprises any feature of new establishment file path, data writing size, access rights.
The default data center selection algorithm of described global metadata server is according to reading or writing data distribution, the systematic function of request of data He Ge data center, any feature of loading condition, adopts distribute preferential, performance preference strategy of data to select data center.
The default dispatching algorithm of described metadata node comprises any feature according to the distance of size of data, piecemeal quantity, data block and client, data block distribution, by distance priority, distribution fairness policy, selects.
The HDFS data read-write system of many data centers of the present invention adopts two-layer logical separation scheduling architecture.Global logic layer has global metadata server to be responsible for the distribution of each data center's NameSpace, the selection of the inquiry of global metadata, the data center while reading and writing data is by the integrated unified core in each independent data center.Business Logic is by carrying out regarded as output controlling to the metadata node of HDFS, increase GMS plugin module, and link as slave module with global metadata server, thereby form, can support metadata synchronization and many HDFS data center resource-sharing framework of sharing.The present invention realizes the overall situation of metadata and shares in the functions such as metadata node data management of the original HDFS of reservation data center.This mode reduces system complexity when keeping original system efficient stable, can effectively realize fast the reading and writing data access of many HDFS data center.
Accompanying drawing explanation
Fig. 1 is HDFS system architecture diagram.
Fig. 2 is the HDFS data read-write system Organization Chart of the many data centers of the present invention.
Fig. 3 is the HDFS read data flow process figure of the many data centers of the present invention.
Fig. 4 is that the HDFS of the many data centers of the present invention writes data flowchart.
Embodiment
In order to be illustrated more clearly in technical scheme of the present invention, below in conjunction with the drawings and specific embodiments, describe the present invention.
As shown in Figure 2, a kind of HDFS data read-write system of many data centers, comprise a global metadata server (Global Metadata Server, GMS), be numbered 01 to N n data center, a client Client, all there are a metadata node (NameNode) and a plurality of back end (DataNode) in each data center, wherein between global metadata server and client, adopt wide area network to link, between the metadata node of global metadata server He Ge data center, adopt wide area network link, between the metadata node of each data center and back end, by local area network (LAN), linked.Global metadata server, for the metadata information of the store and management overall situation, is responsible for each data center and distributes metadata NameSpace; In the metadata node of each data center, all comprise a GMS plugin (global metadata server middleware) module, and link with global metadata server, to global metadata server registration timing, report data center's resource using status and metadata information.
Global metadata server is responsible for receiving client HDFS reading and writing data access request, and selects according to default dispatching algorithm the data center meeting the demands; The metadata node at the above-mentioned selected data of client-access center, by this metadata node, carried out the scheduling of HDFS reading and writing data, after client HDFS reading and writing data completes, the metadata node of data center is synchronized to global metadata server by the change information of metadata again.
Global metadata server is for the metadata information of the store and management overall situation; Be responsible for each data center and distribute metadata NameSpace; Be responsible for receiving client HDFS reading and writing data access request, and select according to default dispatching algorithm metadata node corresponding to data center meeting the demands; Be responsible for the metadata updates of the metadata node of each data center of reception.
Global metadata server is mainly comprised of access interface, GMS service routine, three templates of metadata management; Access interface is the interactive interface module of client and global metadata server, is responsible for processing client to requests such as the reading and writing of HDFS data, inquiries; GMS service routine is that module is guarded in the service of global metadata server, and operation monitoring, the module of being responsible for global metadata server are restarted, and guarantees the stable operation of global metadata server; The metadata node of metadata management Shi Ge data center and the interactive interface module of global metadata server, be in charge of the metadata node of each data center, receive the metadata synchronization update request of each data center and store global metadata information, process the reading and writing data request that access interface module receives, and according to the suitable data center of condition selecting of global metadata information He Ge data center.
GMS plugin module, is a middleware for communicating with global metadata server, is responsible for to global metadata server registration, in real time to global metadata server sync notebook data center situation information and metadata information.
The metadata node of each data center (containing GMS plugin module), for managing directory tree and the file metadata information at notebook data center, is passed through GMS plugin module real-time synchronization information to global metadata server by Predistribution Algorithm when the metadata of metadata node changes; Metadata node is responsible for the management of the back end at notebook data center, the processing of client HDFS reading and writing data request of data; Metadata node can, according to data parameters to be visited and default scheduling strategy, be selected back end from the data center of its management.
The back end of each data center is for the read-write of the storage on management node, piece list, data; Back end under the scheduling of metadata node, carry out piece establishment, delete and copy; Back end is periodically reported data block information to metadata node by Predistribution Algorithm.
Client is used for and system interaction, and client is carried out the piecemeal of data writing, and mutual with metadata node, the back end of global metadata server and data center respectively, completes HDFS data read-write operation.
Optional 1 to 200 of the HDFS data read-write system number n of Tu2Duo data center.
By Fig. 2 system, the present invention also provides the HDFS data of Liao Duo data center to read method, below by Fig. 3, is described:
S301 sets up global metadata server, for the metadata information of the store and management overall situation; Global metadata server Wei Ge HDFS data center distributes NameSpace, and metadata information is reported to global metadata server by each data center;
S302 global metadata server receives client HDFS read data request, selects to meet the HDFS data center of reading requirement by preset algorithm, returns to the metadata node information at selected data center;
Client read data request comprises the information such as file path, data block index, buffer size;
Preset dispatching algorithm basis and read the information such as the data distribution of HDFS request of data He Ge data center, systematic function, loading condition, employing data distribute preferentially, performance priority scheduling policy selection data center;
The metadata node of S303 client-access HDFS data center, metadata node returns to client according to default dispatching algorithm data block and back end information;
Metadata node comprises according to information such as the distance of data block and client, piecemeal quantity, data block distributions and provides recommendation reading order according to default dispatching algorithm, by distance priority, distribution fairness policy, select, also can develop as required customization by those skilled in the art;
S304 client and back end carry out alternately, and reading out data has read rear notice metadata node, and read data has operated.
By Fig. 2 system, the invention provides the HDFS data write method of many data centers, below by Fig. 4, be described:
S401 sets up global metadata server, for the metadata information of the store and management overall situation; Global metadata server Wei Ge data center distributes NameSpace, and metadata information is reported to global metadata server by each data center;
S402 global metadata server receives client read data request, by preset algorithm, selects to meet the HDFS data center that writes requirement, returns to the metadata node information of selected HDFS data center;
Client write data requests comprises the information such as new establishment file path, data writing size, access rights;
The default dispatching algorithm of global metadata server is according to concrete data centers of Information Selection such as the data distribution of solicited message He Ge data center, systematic function, loading conditions, adopt distribute preferential, performance priority scheduling strategy of data to dispatch, dispatching algorithm can be by those skilled in the art's flexible customization as required;
The metadata node of the selected HDFS of S403 client-access data center, metadata node creates metadata information, and according to default dispatching algorithm distribute data node, and back end information is returned to client;
The default dispatching algorithm of metadata node comprises according to information such as size of data, piecemeal quantity, data block distributions, by distance priority, the strategies such as justice that distribute, dispatches, and also can develop as required customization by those skilled in the art;
S404 client and back end carry out alternately, carrying out data writing operation, have write rear notice metadata node; During client data writing, adopt piecemeal writing mechanism, data block copy copy is completed automatically by back end, and all data blocks all write successfully notifies metadata node to write afterwards;
S405 is after ablation process completes, and the metadata node of HDFS data center is synchronized to global metadata server by the change information of metadata.
In sum, the invention solves along with disperseing the middle-size and small-size data center of independence everywhere more and more, and the storage resources and the data that are difficult to realize each data center are effectively shared, and the unified problems such as data access interface are provided how to outer layer service, unified management, unified interface have been realized, HDFS reading and writing data framework and the method for many data centers of open and stable.
Claims (5)
- The HDFS data read-write system of 1.Yi Zhongduo data center, is characterized in that, comprises a global metadata server, a n data center, a client, and all there are a metadata node and a plurality of back end in each data center; Between global metadata server and client He Ge data center metadata node, adopt wide area network to link, between the metadata node of each data center and back end, by local area network (LAN), linked; Global metadata server, for the metadata information of the store and management overall situation, is responsible for each data center and distributes metadata NameSpace; The metadata node of each data center all comprises a GMS plugin module, is responsible for to global metadata server registration and regularly reports data center's resource using status and metadata information; Global metadata server is responsible for receiving client HDFS reading and writing data access request, and selects according to default dispatching algorithm the data center meeting the demands; The metadata node at client-access selected data center, by this metadata node, carried out the scheduling of HDFS reading and writing data, client is after HDFS reading and writing data completes, and the metadata node of data center is synchronized to global metadata server by the change information of metadata again.
- The HDFS data read-write method of 2.Yi Zhongduo data center, is characterized in that, comprises the large step of read and write two:The first step, HDFS data are read, and comprising:(1) set up global metadata server, for the metadata information of the store and management overall situation; Global metadata server Wei Ge data center distributes NameSpace, and metadata information is reported to global metadata server by each data center;(2) global metadata server receives client read data request, selects to meet the data center of reading requirement by preset algorithm, returns to the metadata node information at selected data center;(3) metadata node of client-access data center, metadata node returns to client according to default dispatching algorithm data block and back end information;(4) client and back end carry out alternately, and reading out data has read rear notice metadata node, and read data has operated;Second step, HDFS data are write, and comprising:(1) step of reading with HDFS data (1);(2) global metadata server receives client read data request, by preset algorithm, selects to meet the data center that writes requirement, returns to the metadata node information at selected data center;(3) metadata node of the selected HDFS of client-access data center, metadata node creates metadata information, and according to preset algorithm distribute data node, and back end information is returned to client;(4) client and back end carry out alternately, carrying out data writing operation, have write rear notice metadata node; During client data writing, adopt piecemeal writing mechanism, data block copy copy is completed automatically by back end, and all data blocks all write successfully notifies metadata node to write afterwards;(5), after ablation process completes, the metadata node of data center is synchronized to global metadata server by the change information of metadata.
- 3. the HDFS data read-write method of many data centers as claimed in claim 2, is characterized in that, described client read data request comprises any feature of file path, data block index, buffer size; Described client write data requests comprises any feature of new establishment file path, data writing size, access rights.
- 4. the HDFS data read-write method of many data centers as claimed in claim 2, it is characterized in that, the default data center selection algorithm of described global metadata server is according to reading or writing data distribution, the systematic function of request of data He Ge data center, any feature of loading condition, adopts distribute preferential, performance preference strategy of data to select data center.
- 5. the HDFS data read-write method of many data centers as claimed in claim 2, it is characterized in that, the default dispatching algorithm of described metadata node comprises any feature according to the distance of size of data, piecemeal quantity, data block and client, data block distribution, by distance priority, distribution fairness policy, selects.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410344218.9A CN104113597B (en) | 2014-07-18 | 2014-07-18 | The HDFS data read-write method of a kind of many Data centres |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410344218.9A CN104113597B (en) | 2014-07-18 | 2014-07-18 | The HDFS data read-write method of a kind of many Data centres |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104113597A true CN104113597A (en) | 2014-10-22 |
| CN104113597B CN104113597B (en) | 2016-06-08 |
Family
ID=51710229
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410344218.9A Active CN104113597B (en) | 2014-07-18 | 2014-07-18 | The HDFS data read-write method of a kind of many Data centres |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104113597B (en) |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104506527A (en) * | 2014-12-23 | 2015-04-08 | 苏州海博智能系统有限公司 | Multidimensional information pointer platform and data access method thereof |
| CN105049504A (en) * | 2015-07-09 | 2015-11-11 | 国云科技股份有限公司 | Big data transit transmission synchronization and storage method |
| CN105760556A (en) * | 2016-04-19 | 2016-07-13 | 江苏物联网研究发展中心 | Low-time delay high-throughput multi-copy file read-write optimization method |
| CN105847392A (en) * | 2016-04-25 | 2016-08-10 | 乐视控股（北京）有限公司 | HDFS writing method and device |
| CN106357723A (en) * | 2016-08-15 | 2017-01-25 | 杭州古北电子科技有限公司 | Synchronous system and method for multi-cluster information caching based on cloud host |
| CN106502795A (en) * | 2016-11-03 | 2017-03-15 | 郑州云海信息技术有限公司 | The method and system of scientific algorithm application deployment are realized on distributed type assemblies |
| WO2017206754A1 (en) * | 2016-05-30 | 2017-12-07 | 中兴通讯股份有限公司 | Storage method and storage device for distributed file system |
| CN107483571A (en) * | 2017-08-08 | 2017-12-15 | 柏域信息科技（上海）有限公司 | A kind of dynamic cloud storage method and system |
| CN107562926A (en) * | 2017-09-14 | 2018-01-09 | 丙申南京网络技术有限公司 | For more hadoop distributed file systems of big data analysis |
| CN107958159A (en) * | 2017-11-15 | 2018-04-24 | 广东电网有限责任公司电力调度控制中心 | A kind of method and system of big data migration |
| CN109582686A (en) * | 2018-12-13 | 2019-04-05 | 中山大学 | Distributed meta-data management consistency ensuring method, device, system and application |
| CN109726250A (en) * | 2018-12-27 | 2019-05-07 | 星环信息科技（上海）有限公司 | Data-storage system, metadatabase synchronization and data cross-domain calculation method |
| CN110022338A (en) * | 2018-01-09 | 2019-07-16 | 阿里巴巴集团控股有限公司 | File reading, system, meta data server and user equipment |
| CN110213352A (en) * | 2019-05-17 | 2019-09-06 | 北京航空航天大学 | The unified Decentralized Autonomous storage resource polymerization of name space |
| CN111030858A (en) * | 2019-12-06 | 2020-04-17 | 北京浪潮数据技术有限公司 | Data management method, system and related device for distributed multi-cluster system |
| CN111124301A (en) * | 2019-12-18 | 2020-05-08 | 深圳供电局有限公司 | Data consistency storage method and system of object storage device |
| CN111198849A (en) * | 2020-01-10 | 2020-05-26 | 国网福建省电力有限公司 | Power supply data read-write system based on Hadoop and working method thereof |
| CN111327681A (en) * | 2020-01-21 | 2020-06-23 | 北京工业大学 | Cloud computing data platform construction method based on Kubernetes |
| CN112395354A (en) * | 2020-11-05 | 2021-02-23 | 深圳市中博科创信息技术有限公司 | Distributed relational database based on HDFS metadata server and construction method |
| CN113419687A (en) * | 2021-07-13 | 2021-09-21 | 广东电网有限责任公司 | Object storage method, system, equipment and storage medium |
| US20220075757A1 (en) * | 2019-09-27 | 2022-03-10 | Huawei Technologies Co., Ltd. | Data read method, data write method, and server |
| CN117076391A (en) * | 2023-10-12 | 2023-11-17 | 长江勘测规划设计研究有限责任公司 | Water conservancy metadata management system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102419766A (en) * | 2011-11-01 | 2012-04-18 | 西安电子科技大学 | Data redundancy and file operation methods based on Hadoop distributed file system (HDFS) |
| CN103473365A (en) * | 2013-09-25 | 2013-12-25 | 北京奇虎科技有限公司 | File storage method and device based on HDFS (Hadoop Distributed File System) and distributed file system |
| US20140122429A1 (en) * | 2012-10-31 | 2014-05-01 | International Business Machines Corporation | Data processing method and apparatus for distributed systems |
-
2014
- 2014-07-18 CN CN201410344218.9A patent/CN104113597B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102419766A (en) * | 2011-11-01 | 2012-04-18 | 西安电子科技大学 | Data redundancy and file operation methods based on Hadoop distributed file system (HDFS) |
| US20140122429A1 (en) * | 2012-10-31 | 2014-05-01 | International Business Machines Corporation | Data processing method and apparatus for distributed systems |
| CN103473365A (en) * | 2013-09-25 | 2013-12-25 | 北京奇虎科技有限公司 | File storage method and device based on HDFS (Hadoop Distributed File System) and distributed file system |
Cited By (32)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104506527B (en) * | 2014-12-23 | 2021-12-17 | 苏州海博智能系统有限公司 | Multi-dimensional information pointer platform and data access method thereof |
| CN104506527A (en) * | 2014-12-23 | 2015-04-08 | 苏州海博智能系统有限公司 | Multidimensional information pointer platform and data access method thereof |
| CN105049504A (en) * | 2015-07-09 | 2015-11-11 | 国云科技股份有限公司 | Big data transit transmission synchronization and storage method |
| CN105049504B (en) * | 2015-07-09 | 2019-03-05 | 国云科技股份有限公司 | A kind of big data transfer transmission is synchronous and storage method |
| CN105760556A (en) * | 2016-04-19 | 2016-07-13 | 江苏物联网研究发展中心 | Low-time delay high-throughput multi-copy file read-write optimization method |
| CN105760556B (en) * | 2016-04-19 | 2019-05-24 | 江苏物联网研究发展中心 | More wave files of low delay high-throughput read and write optimization method |
| CN105847392A (en) * | 2016-04-25 | 2016-08-10 | 乐视控股（北京）有限公司 | HDFS writing method and device |
| WO2017206754A1 (en) * | 2016-05-30 | 2017-12-07 | 中兴通讯股份有限公司 | Storage method and storage device for distributed file system |
| CN107451138A (en) * | 2016-05-30 | 2017-12-08 | 中兴通讯股份有限公司 | A kind of distributed file system storage method and system |
| CN106357723A (en) * | 2016-08-15 | 2017-01-25 | 杭州古北电子科技有限公司 | Synchronous system and method for multi-cluster information caching based on cloud host |
| CN106502795A (en) * | 2016-11-03 | 2017-03-15 | 郑州云海信息技术有限公司 | The method and system of scientific algorithm application deployment are realized on distributed type assemblies |
| CN107483571A (en) * | 2017-08-08 | 2017-12-15 | 柏域信息科技（上海）有限公司 | A kind of dynamic cloud storage method and system |
| CN107562926A (en) * | 2017-09-14 | 2018-01-09 | 丙申南京网络技术有限公司 | For more hadoop distributed file systems of big data analysis |
| CN107562926B (en) * | 2017-09-14 | 2023-09-26 | 丙申南京网络技术有限公司 | Multi-hadoop distributed file system for big data analysis |
| CN107958159A (en) * | 2017-11-15 | 2018-04-24 | 广东电网有限责任公司电力调度控制中心 | A kind of method and system of big data migration |
| CN110022338A (en) * | 2018-01-09 | 2019-07-16 | 阿里巴巴集团控股有限公司 | File reading, system, meta data server and user equipment |
| CN109582686A (en) * | 2018-12-13 | 2019-04-05 | 中山大学 | Distributed meta-data management consistency ensuring method, device, system and application |
| CN109582686B (en) * | 2018-12-13 | 2021-01-15 | 中山大学 | Method, device, system and application for ensuring consistency of distributed metadata management |
| CN109726250A (en) * | 2018-12-27 | 2019-05-07 | 星环信息科技（上海）有限公司 | Data-storage system, metadatabase synchronization and data cross-domain calculation method |
| CN109726250B (en) * | 2018-12-27 | 2020-01-17 | 星环信息科技（上海）有限公司 | Data storage system, metadata database synchronization method and data cross-domain calculation method |
| CN110213352A (en) * | 2019-05-17 | 2019-09-06 | 北京航空航天大学 | The unified Decentralized Autonomous storage resource polymerization of name space |
| US20220075757A1 (en) * | 2019-09-27 | 2022-03-10 | Huawei Technologies Co., Ltd. | Data read method, data write method, and server |
| CN111030858A (en) * | 2019-12-06 | 2020-04-17 | 北京浪潮数据技术有限公司 | Data management method, system and related device for distributed multi-cluster system |
| CN111124301A (en) * | 2019-12-18 | 2020-05-08 | 深圳供电局有限公司 | Data consistency storage method and system of object storage device |
| CN111124301B (en) * | 2019-12-18 | 2024-02-23 | 深圳供电局有限公司 | Data consistency storage method and system of object storage device |
| CN111198849A (en) * | 2020-01-10 | 2020-05-26 | 国网福建省电力有限公司 | Power supply data read-write system based on Hadoop and working method thereof |
| CN111327681A (en) * | 2020-01-21 | 2020-06-23 | 北京工业大学 | Cloud computing data platform construction method based on Kubernetes |
| CN112395354A (en) * | 2020-11-05 | 2021-02-23 | 深圳市中博科创信息技术有限公司 | Distributed relational database based on HDFS metadata server and construction method |
| CN112395354B (en) * | 2020-11-05 | 2022-08-02 | 深圳市中博科创信息技术有限公司 | Distributed relational database based on HDFS metadata server and construction method |
| CN113419687A (en) * | 2021-07-13 | 2021-09-21 | 广东电网有限责任公司 | Object storage method, system, equipment and storage medium |
| CN117076391A (en) * | 2023-10-12 | 2023-11-17 | 长江勘测规划设计研究有限责任公司 | Water conservancy metadata management system |
| CN117076391B (en) * | 2023-10-12 | 2024-03-22 | 长江勘测规划设计研究有限责任公司 | Water conservancy metadata management system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104113597B (en) | 2016-06-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104113597B (en) | The HDFS data read-write method of a kind of many Data centres | |
| US9460185B2 (en) | Storage device selection for database partition replicas | |
| US10459898B2 (en) | Configurable-capacity time-series tables | |
| US9489443B1 (en) | Scheduling of splits and moves of database partitions | |
| US9304815B1 (en) | Dynamic replica failure detection and healing | |
| CN105190533B (en) | Snapshot in situ | |
| US8271455B2 (en) | Storing replication requests for objects in a distributed storage system | |
| EP3040886A1 (en) | Service oriented data management and architecture | |
| CN110447021A (en) | For maintaining the methods, devices and systems of the consistency of metadata and data between data center | |
| CN109314721B (en) | Management of multiple clusters of a distributed file system | |
| CN105190623A (en) | Log record management | |
| CN103067461A (en) | Metadata management system of document and metadata management method thereof | |
| CN103458044A (en) | Metadata sharing management method for multi-storage clusters under wide area network environment | |
| US9983823B1 (en) | Pre-forking replicas for efficient scaling of a distribued data storage system | |
| CN103067488A (en) | Implement method of unified storage | |
| CN105069151A (en) | HBase secondary index construction apparatus and method | |
| US20170351620A1 (en) | Caching Framework for Big-Data Engines in the Cloud | |
| CN109992373B (en) | Resource scheduling method, information management method and device and task deployment system | |
| CN103365740B (en) | A kind of data cold standby method and device | |
| CN102693312A (en) | Flexible transaction management method in key-value store data storage | |
| CN202872848U (en) | Cloud storage terminal equipment based on cloud information and cloud computing services | |
| US8543700B1 (en) | Asynchronous content transfer | |
| Salehian et al. | Comparison of spark resource managers and distributed file systems | |
| JP2015114913A (en) | Storage device, storage system, and data management program | |
| RU2721235C2 (en) | Method and system for routing and execution of transactions |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C53 | Correction of patent of invention or patent application | ||
| CB03 | Change of inventor or designer information |
Inventor after: Dong Bo Inventor after: Ruan Jianfei Inventor after: Zheng Qinghua Inventor after: He Huan Inventor after: Zhang Hanning Inventor after: Zhang Weizhan Inventor before: Dong Bo Inventor before: Zhang Hanning Inventor before: Zheng Qinghua Inventor before: He Huan Inventor before: Zhang Weizhan |
|
| COR | Change of bibliographic data |
Free format text: CORRECT: INVENTOR; FROM: DONG BO ZHANG HANNING ZHENG QINGHUA HE HUAN ZHANG WEIZHAN TO: DONG BO RUANJIANFEI ZHENG QINGHUA HE HUAN ZHANG HANNING ZHANG WEIZHAN |
|
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |