CN102932443A - HDFS (hadoop distributed file system) cluster based distributed cloud storage system - Google Patents
HDFS (hadoop distributed file system) cluster based distributed cloud storage system Download PDFInfo
- Publication number
- CN102932443A CN102932443A CN2012104191579A CN201210419157A CN102932443A CN 102932443 A CN102932443 A CN 102932443A CN 2012104191579 A CN2012104191579 A CN 2012104191579A CN 201210419157 A CN201210419157 A CN 201210419157A CN 102932443 A CN102932443 A CN 102932443A
- Authority
- CN
- China
- Prior art keywords
- user
- file
- storage
- cloud storage
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses an HDFS (hadoop distributed file system) cluster based distributed cloud storage system which comprises a name node and a data node, wherein the name node is responsible for managing file namespaces and client access, and the data node is responsible for managing data storage. The system is characterized in that the name node comprises a registration/certification module and a user customization module, wherein the registration/certification module is used for providing users with a login service and carrying out certification on user login requests; and the user customization module is used for customizing the capacity of a storage space of a login user, and after the user logins successfully, the user sets a storage space allowance reminder. According to the method, the size of a storage space of a user is customized, therefore, an effect of distribution based on need is achieved, and resources are rationally used; the application of distributed storage improves the storage efficiency; and the application of multi-copy storage improves the data reliability of storage.
Description
Technical field
The invention belongs to the cloud technical field of memory, be specifically related to a kind of distributed cloud storage system based on the HDFS cluster.
Background technology
Along with the development of Internet technology, amount of information is explosive increase, and the data storage becomes the key issue of restriction enterprise development gradually.Increasing enterprise begins the data storage separated as project independently and manages.High reliability, high universalizable, high scalability, large capacity, cloud is stored with the incomparable advantages characteristic of conventional data centers, is just becoming the important selection that enterprise realizes raising the efficiency, reducing cost.
Compare with traditional memory device, the cloud storage is not only a hardware, but the system that a plurality of parts such as the network equipment, memory device, server, application software, public visit interface, Access Network and a client-side program form.
The cloud storage does not refer to some concrete equipment, and refers to an aggregate that is made of various memory devices and server the user.The user uses the cloud storage, is not to use some memory devices, but a kind of data access service of using whole cloud storage system to bring.The core of cloud storage is that application software combines with memory device, realizes that by application software memory device is to the transformation of stores service.
At present cloud storage is apparent to user's's (no matter being individual or enterprise) meaning: be stored in data on the network and can access whenever and wherever possible and read (as long as can network), save local storage factor and acquire cost according to the additional hardware that growth brings, substantially need not consider the maintenance issues such as data backup, only need to select suitable cloud storage service provider and pay as required correlative charges to get final product.Use the cloud storage to become a kind of trend as stores service, the research that cloud is stored has broad application prospects.
What traditional cloud storage File storage was adopted is centralised storage, and file is stored in the local file system.The deficiency of this storage mode then is when the same data of needs multiaccess, to need many parts of backups.In addition, for the resource that only is kept in the file system, if there is system's machine of delaying, then can cause loss of data.For these problems, existing solution has user data is saved in server, only has and can network, and the user can access its data anywhere or anytime, need not many backups.But the problem of this existence then is the problem of a data reliability, and the machine in case server is delayed will cause user data loss.The present invention therefore.
Summary of the invention
The object of the invention is to provide a kind of distributed cloud storage system based on the HDFS cluster, has solved the storage of prior art medium cloud storage File and has adopted centralized stores, and the user can not customize, and file transfer is difficult to the problems such as control.
In order to solve these problems of the prior art, technical scheme provided by the invention is:
A kind of distributed cloud storage system based on the HDFS cluster, comprise the title node of being in charge of file name space and client-access and the back end of being responsible for data are stored into the line pipe reason, it is characterized in that described title node comprises authentication registration module and customization module, described authentication registration module is used for providing the user to register and user's logging request being authenticated; Described customization module is used for registered user's storage space volume is customized, and the user arranges the prompting of memory space surplus after the user successfully logins.
Preferably, described user directly uploads download file or uploads download file by the title node to back end to back end by client.
Preferably, when the user carried out upload operation, the user at first connected with the server end of title node, obtained user's remaining space size, if user's remaining space is not enough, then pointed out the memory space surplus, and returned the user and store main interface; Otherwise upload user's file, and prompting memory space surplus.
Preferably, when the user carried out down operation, the user at first connected with server end by client, and the sign that user's transmission is read and file identification are to server end, and server end reads sign, returns to the client file data according to file identification.
The registrable account of user in the technical solution of the present invention is applied for a certain size memory space.But the user user name that succeeds in registration and password login storage system.But the user can check its usage space size and the file of having uploaded, and the user can upload local file to server, and server deposits user data among the HDFS in; The user can download its file that has uploaded onto the server.When user's space is not enough, prompting user.
Technical solution of the present invention adopts the piecemeal storage, accelerates storage efficiency; The many copies of file of storage, the reliability of assurance user data.Compare with traditional document storage system, technical solution of the present invention has user's usability, and the user is the accessor server data whenever and wherever possible, adopts the HDFS distributed memory system to improve the storage efficiency of system.The cloud storage is applied in the storage of subscriber data system, possess the user data reliability, even certain loom in the server stores system has been delayed machine, the user still can obtain correct data.
The present invention compared with prior art has following beneficial effect:
Technical solution of the present invention has compared with prior art realized the customization of user's size, and resource is rationally used in distribution according to need; The user can be under networking situation calling party data whenever and wherever possible, need not to copy everywhere; Use distributed storage to improve storage efficiency; Use many copy storages, improved the data reliability of storage.
Description of drawings
The invention will be further described below in conjunction with drawings and Examples:
Fig. 1 is that technical solution of the present invention is based on the Organization Chart of the distributed cloud storage system of HDFS cluster;
Fig. 2 is that technical solution of the present invention is based on the overview flow chart of the distributed cloud storage system of HDFS cluster;
Fig. 3 is the workflow diagram that the technical solution of the present invention user carries out upload file;
Fig. 4 is the workflow diagram that the technical solution of the present invention user carries out download file;
Fig. 5 is the workflow diagram that the technical solution of the present invention user carries out deleted file.
Embodiment
Below in conjunction with specific embodiment such scheme is described further.Should be understood that these embodiment are not limited to limit the scope of the invention for explanation the present invention.The implementation condition that adopts among the embodiment can be done further adjustment according to the condition of concrete producer, and not marked implementation condition is generally the condition in the normal experiment.
Embodiment
The distributed cloud storage system based on the HDFS cluster that the present embodiment obtains, comprise the title node of being in charge of file name space and client-access and the back end of being responsible for data are stored into the line pipe reason, described title node comprises authentication registration module and customization module, and described authentication registration module is used for providing the user to register and user's logging request being authenticated; Described customization module is used for registered user's storage space volume is customized, and the user arranges the prompting of memory space surplus after the user successfully logins.
As shown in Figure 1, carry out the process of map-reduce data decomposition in the HDFS cluster.When specifically disposing, the implementation step is described below referring to Fig. 2:
1) build cluster, configuration HDFS arranges the Namenode/Datanode node, a host node, and two from node, and the copy number.
2) user's registration: when the user registers, judge mainly whether user name is used, what still adopt is the storage mode of structural data herein, if user name exists, then can not register; If user name does not exist, judge then whether the password that the user inputs has contained capital and small letter, digital alphabet etc., the user creates successfully if meet the requirements then, otherwise the prompting user registration failure.
3) user user name and password login system, background data base connects, and judges whether user's input is correct, if checking correctly then enter user storage space, shows that the user stores main interface, otherwise returns login interface.
4) enter after the user stores the interface, control page jump with the action class, operate execution according to the user, the user can select to check the operations such as individual storage information, upload file, then jumps to the corresponding page.
5) be illustrated in figure 3 as the workflow diagram that the user carries out upload file.Select upload file, at first subscriber's main station and server end connect, and obtain user's remaining space size, if user's remaining space is not enough, then provide corresponding prompting, and return the user and store main interface; Otherwise server reads the user and writes sign, and the spanned file path returns to client, and server end writes HDFS with user data.HDFS is then according to load balancing, choose load in the cluster lower from node, piecemeal storage user data, block size can configure according to actual conditions, usually be set to 64M, after the data of having stored a block size, back up according to number of copies, HDFS at first selects to carry out copy storage with the node of storage node on same frame, and then selects not another node on same frame to store.
6) be illustrated in figure 4 as the workflow diagram that the user carries out download file.Select viewing files, the user can download or delete the user file that it has been uploaded.During download file, client at first connects with server end, the sign that the client transmission is downloaded and file identification are to server end, server end reads sign, file identification is sent to HDFS, HDFS finds the nearest position from node of its storage in host node, file reading returns to the client file data according to file identification.Be illustrated in figure 5 as the workflow diagram that the user carries out deleted file.During deleted file, server end according to file identification, is done a mark to file after reading sign, transfer to the HDFS deleted file.HDFS can detect the heartbeat that each file sends every certain time interval, if heartbeat is designated deletion, then HDFS can delete this document.
The effect that below is the centralised storage of routine in the present embodiment and the prior art compares, and is as shown in table 1.
The effect of table 1 technical solution of the present invention and centralised storage relatively
Above-mentioned example only is explanation technical conceive of the present invention and characteristics, and its purpose is to allow the people who is familiar with technique can understand content of the present invention and according to this enforcement, can not limit protection scope of the present invention with this.All equivalent transformations that Spirit Essence is done according to the present invention or modification all should be encompassed within protection scope of the present invention.
Claims (4)
1. distributed cloud storage system based on the HDFS cluster, comprise the title node of being in charge of file name space and client-access and the back end of being responsible for data are stored into the line pipe reason, it is characterized in that described title node comprises authentication registration module and customization module, described authentication registration module is used for providing the user to register and user's logging request being authenticated; Described customization module is used for registered user's storage space volume is customized, and the user arranges the prompting of memory space surplus after the user successfully logins.
2. the distributed cloud storage system based on the HDFS cluster according to claim 1 is characterized in that described user directly uploads download file or uploads download file by the title node to back end to back end by client.
3. the distributed cloud storage system based on the HDFS cluster according to claim 2, it is characterized in that when the user carries out upload operation, the user at first connects with the server end of title node, obtain user's remaining space size, if user's remaining space is not enough, then point out the memory space surplus, and return the user and store main interface; Otherwise upload user's file, and prompting memory space surplus.
4. the distributed cloud storage system based on the HDFS cluster according to claim 2, it is characterized in that when the user carries out down operation, the user at first connects with server end by client, the sign that user's transmission is read and file identification are to server end, server end reads sign, returns to the client file data according to file identification.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012104191579A CN102932443A (en) | 2012-10-29 | 2012-10-29 | HDFS (hadoop distributed file system) cluster based distributed cloud storage system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012104191579A CN102932443A (en) | 2012-10-29 | 2012-10-29 | HDFS (hadoop distributed file system) cluster based distributed cloud storage system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102932443A true CN102932443A (en) | 2013-02-13 |
Family
ID=47647140
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012104191579A Pending CN102932443A (en) | 2012-10-29 | 2012-10-29 | HDFS (hadoop distributed file system) cluster based distributed cloud storage system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102932443A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103116643A (en) * | 2013-02-25 | 2013-05-22 | 江苏物联网研究发展中心 | Hadoop-based intelligent medical data management method |
| CN103442057A (en) * | 2013-08-27 | 2013-12-11 | 玉林师范学院 | Cloud storage system based on user collaboration cloud |
| CN105898601A (en) * | 2015-12-11 | 2016-08-24 | 乐视网信息技术（北京）股份有限公司 | File play method and device |
| CN107592503A (en) * | 2017-09-22 | 2018-01-16 | 唐山开用网络信息服务有限公司 | Audio-video collection management system of enforcing the law and management method |
| CN107613026A (en) * | 2017-10-31 | 2018-01-19 | 四川仕虹腾飞信息技术有限公司 | Distributed file management system based on cloud storage system |
| CN107741981A (en) * | 2017-10-16 | 2018-02-27 | 桂进林 | A kind of e-book management method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN201657029U (en) * | 2010-04-15 | 2010-11-24 | 王鹏 | Cloud storage system based on cloud computing framework |
| CN102196049A (en) * | 2011-05-31 | 2011-09-21 | 北京大学 | Method suitable for secure migration of data in storage cloud |
| CN102404399A (en) * | 2011-11-18 | 2012-04-04 | 浪潮电子信息产业股份有限公司 | Fuzzy dynamic allocation method for cloud storage resource |
| CN102594899A (en) * | 2011-12-31 | 2012-07-18 | 成都市华为赛门铁克科技有限公司 | Storage service method and storage server using the same |
| US20120182891A1 (en) * | 2011-01-19 | 2012-07-19 | Youngseok Lee | Packet analysis system and method using hadoop based parallel computation |
-
2012
- 2012-10-29 CN CN2012104191579A patent/CN102932443A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN201657029U (en) * | 2010-04-15 | 2010-11-24 | 王鹏 | Cloud storage system based on cloud computing framework |
| US20120182891A1 (en) * | 2011-01-19 | 2012-07-19 | Youngseok Lee | Packet analysis system and method using hadoop based parallel computation |
| CN102196049A (en) * | 2011-05-31 | 2011-09-21 | 北京大学 | Method suitable for secure migration of data in storage cloud |
| CN102404399A (en) * | 2011-11-18 | 2012-04-04 | 浪潮电子信息产业股份有限公司 | Fuzzy dynamic allocation method for cloud storage resource |
| CN102594899A (en) * | 2011-12-31 | 2012-07-18 | 成都市华为赛门铁克科技有限公司 | Storage service method and storage server using the same |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103116643A (en) * | 2013-02-25 | 2013-05-22 | 江苏物联网研究发展中心 | Hadoop-based intelligent medical data management method |
| CN103442057A (en) * | 2013-08-27 | 2013-12-11 | 玉林师范学院 | Cloud storage system based on user collaboration cloud |
| CN105898601A (en) * | 2015-12-11 | 2016-08-24 | 乐视网信息技术（北京）股份有限公司 | File play method and device |
| CN107592503A (en) * | 2017-09-22 | 2018-01-16 | 唐山开用网络信息服务有限公司 | Audio-video collection management system of enforcing the law and management method |
| CN107741981A (en) * | 2017-10-16 | 2018-02-27 | 桂进林 | A kind of e-book management method and device |
| CN107613026A (en) * | 2017-10-31 | 2018-01-19 | 四川仕虹腾飞信息技术有限公司 | Distributed file management system based on cloud storage system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11750607B2 (en) | Identifying accounts having shared credentials | |
| CN104008028B (en) | Intelligent mobile terminal data backup memory method and system based on many cloud storages | |
| US9130922B2 (en) | Using a session continuity token to access an online content management system | |
| CN102904870B (en) | Server unit and information processing method | |
| CN110827097A (en) | Tax management method, apparatus, medium and electronic device based on block chain system | |
| CN102932443A (en) | HDFS (hadoop distributed file system) cluster based distributed cloud storage system | |
| CN103036956A (en) | Filing system and implement method of distributed configured massive data | |
| CN103095848B (en) | The cloud folder arrangement of To enterprises client and the method for information interaction | |
| CN103731489B (en) | A kind of date storage method, system and equipment | |
| CN103180842A (en) | Cloud computing system and data synchronization method therefor | |
| CN103605798A (en) | Method for directly operating file stored at cloud end | |
| CN110191128A (en) | A kind of tax shared file system and implementation method based on HDFS | |
| CN104462185A (en) | Digital library cloud storage system based on mixed structure | |
| CN108776758A (en) | The block level data De-weight method of dynamic ownership management is supported in a kind of storage of mist | |
| US20190354395A1 (en) | Limiting folder and link sharing | |
| CN104333553A (en) | Mass data authority control strategy based on combination of blacklist and whitelist | |
| US20240106902A1 (en) | Communication protocols for an online content management system | |
| US20120047568A1 (en) | Digital Asset Management on the Internet | |
| US9436769B2 (en) | Automatic device upload configuration | |
| CN109951567A (en) | A kind of Double Data center applications dispositions method | |
| US10839090B2 (en) | Digital data processing system for efficiently storing, moving, and/or processing data across a plurality of computing clusters | |
| CN111565144A (en) | Data layered storage management method for instant communication tool | |
| CN113840013B (en) | Document system for hierarchical management | |
| Rongqiang et al. | Sceapi: A unified restful web api for high-performance computing | |
| US10412586B2 (en) | Limited-functionality accounts |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| AD01 | Patent right deemed abandoned |
Effective date of abandoning: 20170111 |
|
| AD01 | Patent right deemed abandoned |