CN105516313A - Distributed storage system used for big data - Google Patents
Distributed storage system used for big data Download PDFInfo
- Publication number
- CN105516313A CN105516313A CN201510910421.2A CN201510910421A CN105516313A CN 105516313 A CN105516313 A CN 105516313A CN 201510910421 A CN201510910421 A CN 201510910421A CN 105516313 A CN105516313 A CN 105516313A
- Authority
- CN
- China
- Prior art keywords
- server
- data
- storage
- server cluster
- application server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1044—Group management mechanisms
Abstract
The invention discloses a distributed storage system used for big data, which comprises an application server cluster consisting of a plurality of application servers, a storage server cluster consisting of a plurality of storage servers, and a metadata server cluster consisting of a plurality of metadata servers, wherein the application servers, the storage servers and the metadata server have data exchange through communication network interconnection; the application server cluster is used for generating and receiving production data, the storage server cluster is used for providing storage space, and the metadata server cluster is used for managing the storage space. The distributed storage system used for big data, which is provided by the invention, has the advantages of simple expansion, low cost, high safety and good compatibility.
Description
Technical field
The present invention relates to Computer Applied Technology field, particularly relate to a kind of distributed memory system for large data.
Background technology
Along with the development of the technology such as cloud computing, Internet of Things, mobile Internet, data volume all sharp increases that human society every field produces, these large data have become a kind of important resource, and define data-centered, to be stored as the development trend at center, therefore, how to store these large data and become extremely important.
Conventional store adopts centralized architecture to carry out data storage, and all data by the fail safe of disk array Raid technique guarantee data, and are centrally stored on special memory device, and all creation datas are all transmitted by network or specific store network.But there is dilatation difficulty in this centralized architecture, cost is high, fail safe is low, the shortcoming of poor compatibility.
Summary of the invention
For above-mentioned technical problem, the object of the present invention is to provide a kind of distributed memory system for large data, can effectively solve the problem.
For reaching this object, the present invention by the following technical solutions:
A kind of distributed memory system for large data, comprise: the metadata server cluster of the application server cluster of some application server compositions, the storage server cluster of some storage server compositions and some meta data server compositions, described application server, storage server and meta data server carry out exchanges data by communication network interconnection; Described application server cluster for generation of with reception creation data, described storage server cluster is used for providing memory space, described metadata server cluster be used for managed storage space.
Especially, described storage server, meta data server all adopt X86-based server, simultaneously supporting document and block agreement.
Especially, described in be stored in each application server, storage server and meta data server data all adopt copy mechanism to carry out data protection, and each copy data is all stored on the node different from maternal data.
Especially, described application server, storage server and meta data server are interconnected by full line speed Infiniband network.
The distributed memory system for large data that the present invention proposes adopts distributed structure/architecture; storage server is supporting document and block agreement simultaneously; all storage servers carry out data storage simultaneously; the data be stored on storage server all adopt copy mechanism to carry out data protection; and each copy data is all stored in the storage server different from maternal data, the mode of all data acquisition front end data sections, rear end distribution stores.Thus dilatation is simple, cost the, advantage that fail safe is high, compatible good to make this storage system have.
Accompanying drawing explanation
Fig. 1 is the distributed memory system structured flowchart for large data that the embodiment of the present invention provides.
Embodiment
Below in conjunction with drawings and Examples, the invention will be further described.Be understandable that, specific embodiment described herein is only for explaining the present invention, but not limitation of the invention.It also should be noted that, for convenience of description, illustrate only part related to the present invention in accompanying drawing but not full content, unless otherwise defined, all technology used herein and scientific terminology are identical with belonging to the implication that those skilled in the art of the present invention understand usually.The object of term used in the description of the invention herein just in order to describe specific embodiment, is not intended to be restriction the present invention.Term as used herein " and/or " comprise arbitrary and all combinations of one or more relevant Listed Items.
Embodiment one
Please refer to shown in Fig. 1, the distributed memory system structured flowchart for large data that Fig. 1 provides for the embodiment of the present invention.
In the present embodiment, the distributed memory system for large data comprises the metadata server cluster of the application server cluster of some application server compositions, the storage server cluster of some storage server compositions and some meta data server compositions.Described application server cluster for generation of with reception creation data, described storage server cluster is used for providing memory space, described metadata server cluster be used for managed storage space.Described storage server adopts x86 IA frame serverPC, is specially the hard disk such as SATA or SAS or SSD; Described meta data server adopts x86 IA frame serverPC, is specially SSD; Described application server adopts supports all IA frame serverPC equipment.Described application server, storage server and meta data server are interconnected by Infiniband, carry out exchanges data.
The data storage procedure of described storage system comprises writing and reading process, is specially:
Ablation process:
1, creation data is produced by application server cluster.
2, application server is to meta data server application write request, and request content comprises the size of the data file of needs write.
3, meta data server inquires about the address space of whole storage system, finds out the address block that can distribute, and the address block information preparing to distribute is turned back to application server.
4, application server is according to the address block information returned, and by file fragmentation, and is transferred to each memory node, writes successfully, and each node returns and is written to function signal to meta data server, and address block space identification is " using " by meta data server.
5, the write of data is completed.
Readout:
1, application server issues the read request to meta data server, and request comprises the file path information of reading.
2, the meta data server address block information of storage server of being correlated with according to file path information inquiry, and information is turned back to application server.
3, application server is according to the address block information of each storage server, sends data block read information to each memory node.
4, corresponding application data block is turned back to application server according to data block information by memory node.
For in the distributed memory system of large data described in the present embodiment, adopt the extensive open cluster storage architecture of band external schema, by Infiniband express network, the CPU in common store server, internal memory, the network bandwidth, disk running speed, bus performance are combined, form a high end storage, make to store the transmission speed of overall performance higher than computing ability and the network bandwidth, shield the memory property bottleneck because disk and bus performance Slow lifting cause with this.With the storage of the computing node of full line speed Infiniband network interconnection front end and rear end and metadata node, reach maximization to ensure that the performance of memory device exports.Adopt band external schema, well eliminate the mutual interference of metadata path and data path.And the clustering of meta data server and the clustering of storage server, make the performance bottleneck point do not existed in whole system as NAS or SAN system, the high concurrent demand of high-performance calculation application can be met completely.
Simultaneously, on the basis of unrestricted online dilatation and load-balancing technique fast, Highly Scalable memory technology is adopted for the distributed memory system of large data described in the present embodiment, make the scale of storage system can efficient along with the development of computing application, without the need to the online expansion of break in service, reach the linear growth of the increase performance along with capacity simultaneously.
Simultaneously, for the basis of the distributed memory system of large data multivariate data server Clustering in distributed storage described in the present embodiment, create in conjunction with high-performance calculation application file, access and bibliographic structure feature, adopt multivariate data server Clustering, dynamic-configuration can be carried out according to the scale producing quantity of documents actual in user's production system to metadata server cluster in a set of storage system, meta data servers all in whole storage system participates in retrieval and the positioning work of file simultaneously, eliminate the bottleneck of metadata channel in storage system, simultaneously can also according to the enhancing of the change of subsequent applications feature and demand, accomplish online extended metadata server, and reach the effect of plug and play.
Meanwhile, multistage intelligent Data cache technology is adopted to realize efficient buffer memory for the distributed memory system of large data described in the present embodiment.First, by independent research client kernel module on application server, the feature of analytical applications visit data, adopt specific cache algorithm on the application server valid cache application visit data; Secondly, the cache pool that formation one of the internal storage set of all storage servers being got up is large, by the data of cache algorithm buffer memory different application access uniquely on storage server; Finally, the efficient metadata cache pond of formation one of the internal storage set of all meta data servers being got up, carrys out cache metadata information by the metadata life cycle algorithm of complexity.By the caching technology of such multi-stage data, the performance of whole storage system can be made to be largely increased.
Simultaneously, enhancement mode rights management techniques is provided for the distributed memory system of large data described in the present embodiment, except the original rights management based on operating system user of compatibility, also add enhancement mode rights management techniques specific to self, by the rear end pipe platform of storage system directly for certain catalogue specifies which user or which application server to access, which catalogue forbids that user deletes, amendment etc.
Meanwhile, be that storage server or meta data server all have employed clustering technology for the distributed memory system of large data described in the present embodiment; Further, storage system software layer adopts multi-duplicate technology, file Dynamic Program Slicing technology, file-level raid technology and supports that different directories arranges different redundancy technics to ensure the High Availabitity of whole system.Finally, by these efficient total system scale data High Availabitity technology, thoroughly eliminate the Single Point of Faliure in storage system.In addition, automatic fault finding mechanism that storage system is built-in, can the different fault of dynamic instrumentation, and the recovery flow process of log-on data guarantees data integrity automatically, date restoring is held completely after storing and is carried out, front end applications is not had any impact, the operation that the application of user is continual and steady is guaranteed while guaranteeing data security, separate unit storage server or meta data server are damaged and all can not application data read-write have any impact, and along with the continuous increase of storage server, the speed of date restoring is doubled and redoubled, namely the fail safe of the larger data of service cluster scale is higher.
Simultaneously, object storage technology is adopted for the distributed memory system of large data described in the present embodiment, can accomplish to take disk as the online real time access flow control technology of granularity, when whole storage system is by when monitoring finds that some equipment or disk are in the situation of high capacity in real time, on the one hand, storage system by forbidding some equipment write data, thus can reduce the load of some equipment; On the other hand, can start up system load adjusting function, the hot spot data of system moves according to self intelligent load mechanism by storage system, makes each memory node load balancing in storage system.
Simultaneously, file microtomy is adopted for the distributed memory system of large data described in the present embodiment, file is evenly distributed in whole storage server cluster, be no matter access module to Single document or multiple file to ensure to apply, effectively can utilize the performance that whole storage server cluster provides; Application server end is directly communicated with storage server by express network on the other hand, make use of the bandwidth that express network provides efficiently, eliminates the network bottleneck in current process.After adopting such technology, when whole computing platform thousands of application nodes simultaneously visit data time, can guarantee that each equipment of whole storage system can give play to the hardware capabilities of self, for front end applications provides maximum aggregate bandwidth.
Set up described in the present embodiment and for the concrete steps of the distributed memory system of large data be:
1, meta data server is installed
(1) meta data server operating system is prepared.
(2) meta data server software is installed.
(3) Metadata Service is started.
2, storage server is installed, and associates with meta data server, thus the unified management realized each storage server node and allocation of space
(1) storage server operating system is prepared.
(2) storage server software is installed.
(3) storage data, services is started.
3, Install and configure application server client, realizes the file front end section function of client
(1) application server client is installed.
(2) configure application server client.
Technical scheme of the present invention; distributed memory system for large data adopts distributed structure/architecture; storage server is supporting document and block agreement simultaneously; all storage servers carry out data storage simultaneously; the data be stored on storage server all adopt copy mechanism to carry out data protection; and each copy data is all stored in the storage server different from maternal data, the mode of all data acquisition front end data sections, rear end distribution stores.Thus dilatation is simple, cost the, advantage that fail safe is high, compatible good to make this storage system have.
Note, above are only preferred embodiment of the present invention and institute's application technology principle.Skilled person in the art will appreciate that and the invention is not restricted to specific embodiment described here, various obvious change can be carried out for a person skilled in the art, readjust and substitute and can not protection scope of the present invention be departed from.Therefore, although be described in further detail invention has been by above embodiment, the present invention is not limited only to above embodiment, when not departing from the present invention's design, can also comprise other Equivalent embodiments more, and scope of the present invention is determined by appended right.
Claims (4)
1. the distributed memory system for large data, it is characterized in that, comprise: the metadata server cluster of the application server cluster of some application server compositions, the storage server cluster of some storage server compositions and some meta data server compositions, described application server, storage server and meta data server carry out exchanges data by communication network interconnection; Described application server cluster for generation of with reception creation data, described storage server cluster is used for providing memory space, described metadata server cluster be used for managed storage space.
2. the distributed memory system for large data according to claim 1, is characterized in that, described storage server, meta data server all adopt X86-based server, simultaneously supporting document and block agreement.
3. the distributed memory system for large data according to claim 1; it is characterized in that; the described data being stored in each application server, storage server and meta data server all adopt copy mechanism to carry out data protection, and each copy data is all stored on the server different from maternal data.
4. the distributed memory system for large data according to claim 1, is characterized in that, described application server, storage server and meta data server are interconnected by full line speed Infiniband network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510910421.2A CN105516313A (en) | 2015-12-10 | 2015-12-10 | Distributed storage system used for big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510910421.2A CN105516313A (en) | 2015-12-10 | 2015-12-10 | Distributed storage system used for big data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105516313A true CN105516313A (en) | 2016-04-20 |
Family
ID=55723943
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510910421.2A Pending CN105516313A (en) | 2015-12-10 | 2015-12-10 | Distributed storage system used for big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105516313A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108153759A (en) * | 2016-12-05 | 2018-06-12 | 中国移动通信集团公司 | A kind of data transmission method of distributed data base, middle tier server and system |
| CN108769137A (en) * | 2018-05-08 | 2018-11-06 | 北京初志科技有限公司 | Distributed structure/architecture data storing and reading method and device based on multigroup framework |
| CN109033130A (en) * | 2018-06-04 | 2018-12-18 | 温州市图盛科技有限公司 | A kind of block chain electric power data storage system |
| WO2019033368A1 (en) * | 2017-08-17 | 2019-02-21 | 深圳市优品壹电子有限公司 | Data storage method and device |
| CN112380067A (en) * | 2020-11-30 | 2021-02-19 | 四川大学华西医院 | Metadata-based big data backup system and method under Hadoop environment |
| CN113010229A (en) * | 2021-03-25 | 2021-06-22 | 中国电子科技集团公司第二十九研究所 | Distributed heterogeneous software cluster configuration management framework |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1652090A (en) * | 2005-02-23 | 2005-08-10 | 北京邦诺存储科技有限公司 | Data managing method for network storage system and network storage system constituted thereby |
| CN102833346A (en) * | 2012-09-06 | 2012-12-19 | 上海海事大学 | Storage metadata based security protection system and method for cloud sensitive data |
| US20130151685A1 (en) * | 2011-12-07 | 2013-06-13 | Citrix Systems, Inc. | Controlling A Network Interface Using Virtual Switch Proxying |
| CN104731516A (en) * | 2013-12-18 | 2015-06-24 | 腾讯科技（深圳）有限公司 | Method and device for accessing files and distributed storage system |
-
2015
- 2015-12-10 CN CN201510910421.2A patent/CN105516313A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1652090A (en) * | 2005-02-23 | 2005-08-10 | 北京邦诺存储科技有限公司 | Data managing method for network storage system and network storage system constituted thereby |
| US20130151685A1 (en) * | 2011-12-07 | 2013-06-13 | Citrix Systems, Inc. | Controlling A Network Interface Using Virtual Switch Proxying |
| CN102833346A (en) * | 2012-09-06 | 2012-12-19 | 上海海事大学 | Storage metadata based security protection system and method for cloud sensitive data |
| CN104731516A (en) * | 2013-12-18 | 2015-06-24 | 腾讯科技（深圳）有限公司 | Method and device for accessing files and distributed storage system |
Non-Patent Citations (2)
| Title |
|---|
| 席岩: "为移动互联网而生：数字媒体行业云存储发展研究", 《广播电视信息》 * |
| 范忠磊: "非对称集群存储系统的块虚拟化方法", 《微电子学与计算机》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108153759A (en) * | 2016-12-05 | 2018-06-12 | 中国移动通信集团公司 | A kind of data transmission method of distributed data base, middle tier server and system |
| CN108153759B (en) * | 2016-12-05 | 2021-07-09 | 中国移动通信集团公司 | Data transmission method of distributed database, intermediate layer server and system |
| WO2019033368A1 (en) * | 2017-08-17 | 2019-02-21 | 深圳市优品壹电子有限公司 | Data storage method and device |
| CN108769137A (en) * | 2018-05-08 | 2018-11-06 | 北京初志科技有限公司 | Distributed structure/architecture data storing and reading method and device based on multigroup framework |
| CN109033130A (en) * | 2018-06-04 | 2018-12-18 | 温州市图盛科技有限公司 | A kind of block chain electric power data storage system |
| CN112380067A (en) * | 2020-11-30 | 2021-02-19 | 四川大学华西医院 | Metadata-based big data backup system and method under Hadoop environment |
| CN112380067B (en) * | 2020-11-30 | 2023-08-22 | 四川大学华西医院 | Metadata-based big data backup system and method in Hadoop environment |
| CN113010229A (en) * | 2021-03-25 | 2021-06-22 | 中国电子科技集团公司第二十九研究所 | Distributed heterogeneous software cluster configuration management framework |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9729659B2 (en) | Caching content addressable data chunks for storage virtualization | |
| US9652405B1 (en) | Persistence of page access heuristics in a memory centric architecture | |
| CN105516313A (en) | Distributed storage system used for big data | |
| US9251003B1 (en) | Database cache survivability across database failures | |
| US11061770B1 (en) | Reconstruction of logical pages in a storage system | |
| US10853193B2 (en) | Database system recovery using non-volatile system memory | |
| US11625169B2 (en) | Efficient token management in a storage system | |
| Fu et al. | Performance optimization for managing massive numbers of small files in distributed file systems | |
| US11334523B2 (en) | Finding storage objects of a snapshot group pointing to a logical page in a logical address space of a storage system | |
| CN106066890A (en) | A kind of distributed high-performance data storehouse integrated machine system | |
| CN103595799A (en) | Method for achieving distributed shared data bank | |
| Shankar et al. | High-performance and resilient key-value store with online erasure coding for big data workloads | |
| US11210230B2 (en) | Cache retention for inline deduplication based on number of physical blocks with common fingerprints among multiple cache entries | |
| Zhou et al. | Hierarchical consistent hashing for heterogeneous object-based storage | |
| US11366601B2 (en) | Regulating storage device rebuild rate in a storage system | |
| Al Maruf et al. | Memory disaggregation: advances and open challenges | |
| Xu et al. | YuruBackup: a space-efficient and highly scalable incremental backup system in the cloud | |
| Takatsu et al. | PPFS: A scale-out distributed file system for post-petascale systems | |
| Duan et al. | Gengar: an RDMA-based distributed hybrid memory pool | |
| Kaseb et al. | Redundant independent files (RIF): a technique for reducing storage and resources in big data replication | |
| US11586353B2 (en) | Optimized access to high-speed storage device | |
| Zhou | Large scale distributed file system survey | |
| Keeton et al. | Persistent Memory: a new tier or storage replacement | |
| Chandrasekar et al. | An exploration into object storage for exascale supercomputers | |
| Arteaga et al. | Towards scalable application checkpointing with parallel file system delegation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160420 |
|
| RJ01 | Rejection of invention patent application after publication |