CN104902022B - Distributed file acquisition method and distributed file acquisition system - Google Patents
Distributed file acquisition method and distributed file acquisition system Download PDFInfo
- Publication number
- CN104902022B CN104902022B CN201510280153.0A CN201510280153A CN104902022B CN 104902022 B CN104902022 B CN 104902022B CN 201510280153 A CN201510280153 A CN 201510280153A CN 104902022 B CN104902022 B CN 104902022B
- Authority
- CN
- China
- Prior art keywords
- user
- download
- file
- hadoop
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention provides a distributed file acquisition method and a distributed file acquisition system, wherein the method comprises the following steps: the client sends a file downloading request to a Hadoop file system; mapping a file list to be downloaded by a user to the user from a Hadoop file system; and returning a file list to be downloaded by the user to the client. The step of the client sending a file downloading request to the Hadoop file system further comprises: the client sends a file downloading request to a downloading interface; the download interface sends a permission verification request to a permission verification module; the authority verification module verifies whether the user has the downloading authority; if the download authority exists, the execution is continued; otherwise, ending. The distributed file acquisition method and the distributed file acquisition system provided by the invention do not need to occupy too much network bandwidth and Hadoop resources, so that the Hadoop resources can be saved.
Description
Technical field
The invention belongs to distributed data processing field more particularly to a kind of distributed document acquisition methods and distribution text
Part obtains system.
Background technique
Hadoop distributed file system (Hadoop Distributed File System, HDFS) is a kind of suitable fortune
Distributed file system of the row on common hardware (commodity hardware).The data that HDFS can provide high-throughput are visited
It asks, the application being very suitable on large-scale dataset.For external client, HDFS is just as a traditional hierarchial file structure system
System.It can create, delete, moving or Rename file, etc..The framework of HDFS is constructed based on one group of specific node,
These nodes include NameNode (only one), and Metadata Service is provided inside HDFS；DataNode provides for HDFS
Memory block.Wherein, NameNode is the software run on an independent machine usually in HDFS example.It is responsible for management text
The access of part system name space and control external client.NameNode decides whether will be on File Mapping to DataNode
In copy block.For most common 3 copy blocks, first copy block is stored on the different nodes of same rack, last
A copy block is stored on some node of different racks.NameNode stores all about file system name in one file
Claim the information in space.This file and a record file comprising all affairs will be stored in the local file system of NameNode
On system.
Existing file acquisition method is the file of the HDFS file system directories of timing scan Hadoop, downloads file
Onto local file system, when user will download file, then file is directly obtained from local file system.Such way
It will cause the waste of Hadoop resource rather.
Therefore, better file acquisition method how is provided, becomes technical staff's problem in need of consideration.
Summary of the invention
Technical problem to be solved by the invention is to provide a kind of distributed document acquisition methods and distributed document to obtain
System saves Hadoop resource.
In order to solve the above-mentioned technical problems, the present invention provides a kind of distributed document acquisition methods, comprising:
Client sends file download request to Hadoop file system；
The listed files that user to be downloaded is mapped to user from Hadoop file system；
The listed files to be downloaded of user is returned to client.
As the preferred embodiment of the present invention, the client sends file download request to Hadoop file system
Step further comprises:
Client sends file download request to download interface；
Download interface sending permission checking request is to Authority Verification module；
Whether Authority Verification module verification user has download permission；If there is download permission then continues to execute；Otherwise terminate.
As the preferred embodiment of the present invention, the step of the end, further comprise:
Lack of competence download information is returned to download interface；
Download interface sends unexpected message to client and disconnects the connection with client, terminates.
In order to solve the above-mentioned technical problem, the present invention also provides a kind of distributed documents to obtain system, comprising:
Client modules, for sending file download request to Hadoop file system module；
Hadoop File Mapping module is reflected for the listed files to be downloaded user from Hadoop file system module
It penetrates to user；
Hadoop file system module, for returning to listed files that user to be downloaded to client modules.
As the preferred embodiment of the present invention, the system also includes:
Download interface module for receiving the file download request, sending permission checking request, and receives downloading result
Return to client modules；
Authority Verification module verifies whether user has download permission for receiving the Authority Verification request；If so,
Then send the message for having permission downloading；Correspondingly,
The Hadoop File Mapping module is further used for wanting user when receiving the message for having permission downloading
The listed files of downloading is mapped to user from Hadoop file system.
As the preferred embodiment of the present invention,
The Authority Verification module is further used for returning lack of competence download information to download interface module；
The download interface module is further used for lower transmission unexpected message to client modules and disconnects and client mould
The connection of block.
Distributed document acquisition methods and distributed document provided by the invention obtain system, do not need to occupy too many net
Network bandwidth and Hadoop resource, so as to save Hadoop resource.
Detailed description of the invention
Fig. 1 is the distributed document acquisition methods flow chart of one embodiment of the invention.
Fig. 2 is the distributed document acquisition methods flow chart of another embodiment of the invention.
Fig. 3 is the distributed file system structural schematic diagram of one embodiment of the invention.
Fig. 4 is the distributed file system structural schematic diagram of another embodiment of the invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that the described embodiment is only a part of the embodiment of the present invention, rather than all.Based in the present invention
Embodiment, every other embodiment obtained by those of ordinary skill in the art without making creative efforts, all
Belong to the scope of protection of the invention.
Main idea is that when user will download file, by the HDFS file of Hadoop by way of mapping
Some catalogue of system is mapped to local file system, similar shortcut.In this case there is no need to occupy more Netowrk tapes
Wide and Hadoop resource.
It is the distributed document acquisition methods flow chart of one embodiment of the invention shown in referring to Fig.1.The method packet
It includes:
101, client sends file download request to Hadoop file system；
102, the listed files that user to be downloaded is mapped to user from Hadoop file system；
103, the listed files to be downloaded of user is returned to client.
Referring to shown in Fig. 2, for the distributed document acquisition methods flow chart of another embodiment of the invention.The method packet
It includes:
201, client sends file download request to download interface；
202, download interface sending permission checking request to Authority Verification module；
203, whether Authority Verification module verification user has download permission；If so, thening follow the steps 204, otherwise execute
Step 208；
204, Authority Verification module, which is sent, has permission the message of downloading to Hadoop File Mapping module；
205, the Hadoop File Mapping module listed files to be downloaded user, are mapped to use from Hadoop file system
Family；
206, the listed files that return user needs to download interface；
207, download interface transmission returns results to client, terminates；
208, lack of competence download information is returned to download interface module；
209, download interface module sends unexpected message and disconnects, and terminates.
It is the distributed file system structural schematic diagram of one embodiment of the invention referring to shown in Fig. 3.The system packet
It includes:
Client modules 301, for sending file download request to Hadoop file system module；
Hadoop File Mapping module 302, for the listed files to be downloaded user, from Hadoop file system module
It is mapped to user；
Hadoop file system module 303, for returning to listed files that user to be downloaded to client modules.
Referring to shown in Fig. 4, for the distributed file system structural schematic diagram of another embodiment of the invention.The system packet
It includes:
Client modules 401, for sending file download request；
Download interface module 402 for receiving the file download request, sending permission checking request, and receives downloading
As a result client modules 401 are returned to；
Authority Verification module 403, for verifying whether user has download permission；Downloading is had permission if so, then sending
Otherwise message returns to lack of competence download information to download interface module 402；
Hadoop File Mapping module 404, for receiving the message for having permission downloading；And the text to be downloaded user
Part list is mapped to user from Hadoop file system；
Hadoop file system 405, for returning to the listed files of user's needs to download interface module 402.
In other embodiment of the present invention, the Authority Verification module 403 is further used for returning to information carrying under lack of competence
Cease download interface module 402；
The download interface module 402 is further used for lower transmission unexpected message to client modules 401 and disconnects and visitor
The connection of family end module 401.
Above-described specific embodiment has carried out further the purpose of the present invention, technical scheme and beneficial effects
It is described in detail, it should be understood that being not intended to limit the present invention the foregoing is merely a specific embodiment of the invention
Protection scope, all within the spirits and principles of the present invention, any modification, equivalent substitution, improvement and etc. done should all include
Within protection scope of the present invention.
Claims (2)
1. a kind of distributed document acquisition methods characterized by comprising
Client sends file download request to Hadoop file system, specific sub-step are as follows: client is sent under file
It carries request and arrives download interface, download interface sending permission checking request to Authority Verification module, Authority Verification module verification user
Whether download permission is had: if there is download permission then continues to execute；Otherwise lack of competence download information is returned to download interface, downloading
Interface sends unexpected message to client and disconnects the connection with client, terminates；
The listed files that user to be downloaded is mapped to user from Hadoop file system；
The listed files to be downloaded of user is returned to client.
2. a kind of distributed document obtains system characterized by comprising
Client modules, for sending file download request to Hadoop file system module；
Hadoop File Mapping module is mapped to for the listed files to be downloaded user from Hadoop file system module
User；
Hadoop file system module, for returning to listed files that user to be downloaded to client modules；
Download interface module for receiving the file download request, sending permission checking request, and receives downloading result and returns
To client modules；
Authority Verification module verifies whether user has download permission for receiving the Authority Verification request；If so, then sending out
Send the message for having permission downloading；Correspondingly, the Hadoop File Mapping module, is further used for working as to receive having permission downloading
Message when, the listed files that user to be downloaded is mapped to user from Hadoop file system；
The Authority Verification module is further used for returning lack of competence download information to download interface module；
The download interface module is further used for lower transmission unexpected message to client modules and disconnects and client modules
Connection.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510280153.0A CN104902022B (en) | 2015-05-27 | 2015-05-27 | Distributed file acquisition method and distributed file acquisition system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510280153.0A CN104902022B (en) | 2015-05-27 | 2015-05-27 | Distributed file acquisition method and distributed file acquisition system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104902022A CN104902022A (en) | 2015-09-09 |
| CN104902022B true CN104902022B (en) | 2019-02-26 |
Family
ID=54034418
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510280153.0A Expired - Fee Related CN104902022B (en) | 2015-05-27 | 2015-05-27 | Distributed file acquisition method and distributed file acquisition system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104902022B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110597764B (en) * | 2019-10-10 | 2024-05-07 | 深圳前海微众银行股份有限公司 | File downloading and version management method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102750391A (en) * | 2012-07-06 | 2012-10-24 | 深圳市远行科技有限公司 | File previewing method and system based on Hadoop distribution type |
| CN103279474A (en) * | 2013-04-10 | 2013-09-04 | 深圳康佳通信科技有限公司 | Video file index method and system |
| CN103581190A (en) * | 2013-11-07 | 2014-02-12 | 江南大学 | Method for control over file safety access based on cloud computing technology |
| CN103577500A (en) * | 2012-08-10 | 2014-02-12 | 腾讯科技（深圳）有限公司 | Method for carrying out data processing by distributed file system and distributed file system |
| CN104038771A (en) * | 2014-06-19 | 2014-09-10 | 常州大学 | High-effect streaming media file distributed storage system and method based on Hadoop2 |
-
2015
- 2015-05-27 CN CN201510280153.0A patent/CN104902022B/en not_active Expired - Fee Related
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102750391A (en) * | 2012-07-06 | 2012-10-24 | 深圳市远行科技有限公司 | File previewing method and system based on Hadoop distribution type |
| CN103577500A (en) * | 2012-08-10 | 2014-02-12 | 腾讯科技（深圳）有限公司 | Method for carrying out data processing by distributed file system and distributed file system |
| CN103279474A (en) * | 2013-04-10 | 2013-09-04 | 深圳康佳通信科技有限公司 | Video file index method and system |
| CN103581190A (en) * | 2013-11-07 | 2014-02-12 | 江南大学 | Method for control over file safety access based on cloud computing technology |
| CN104038771A (en) * | 2014-06-19 | 2014-09-10 | 常州大学 | High-effect streaming media file distributed storage system and method based on Hadoop2 |
Non-Patent Citations (2)
| Title |
|---|
| 《基于HDFS的云计算安全技术研究与实现》;郭圣昌;《中国优秀硕士学位论文全文数据库信息科技辑》;20131015;正文第25、32-33页，图3-1 * |
| 郭圣昌.《基于HDFS的云计算安全技术研究与实现》.《中国优秀硕士学位论文全文数据库信息科技辑》.2013,正文第25-33页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104902022A (en) | 2015-09-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2016346890B2 (en) | Selective synchronization and distributed content item block caching for multi-premises hosting of digital content items | |
| CN106537881B (en) | Method and computing equipment for allowing synchronous access to cloud storage system based on stub tracking | |
| CN105045802B (en) | A kind of polymorphic type previewing file system of message-driven | |
| CN103237046B (en) | Support distributed file system and the implementation method of mixed cloud storage application | |
| CN102404338B (en) | File synchronization method and device | |
| WO2017167100A1 (en) | Data migration method and device | |
| CN105740418A (en) | File monitoring and message pushing based real-time synchronization system | |
| CN109542865A (en) | Distributed cluster system configuration file synchronous method, device, system and medium | |
| CN103873290A (en) | Evaluating distributed application performance in a new environment | |
| CN105025053A (en) | Distributed file upload method based on cloud storage technology and system | |
| CN102882985A (en) | File sharing method based on cloud storage | |
| US9847903B2 (en) | Method and apparatus for configuring a communication system | |
| CN106294870B (en) | Object-based distribution cloud storage method | |
| CN109818934A (en) | A kind of method, apparatus and calculating equipment of automation daily record processing | |
| CN102880658A (en) | Distributed file management system based on seismic data processing | |
| CN111400777B (en) | Network storage system, user authentication method, device and equipment | |
| CN104348859B (en) | File synchronisation method, device, server, terminal and system | |
| EP2842034B1 (en) | Providing client and service compatibility through cloud-hosted adapters | |
| CN106953910A (en) | A kind of Hadoop calculates storage separation method | |
| CN104104582B (en) | A kind of data storage path management method, client and server | |
| CN105490843A (en) | Information processing method and system | |
| CN106250571A (en) | The method and system that a kind of ETL data process | |
| CN109525590A (en) | The transmission method and device of data packet | |
| CN114357252A (en) | Storage method, system and storage medium of cross-source multi-domain distributed data | |
| CN101483668A (en) | Network storage and access method, device and system for hot spot data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220428 Address after: 100000 room 116, building 3, Shuangqiao (Shuangqiao dairy factory), Chaoyang District, Beijing Patentee after: Beijing Xiaoxiang innovation Artificial Intelligence Technology Co.,Ltd. Address before: 100085 901, 9th floor, building 5, yard 1, Shangdi East Road, Haidian District, Beijing Patentee before: BEIJING GEO POLYMERIZATION TECHNOLOGY Co.,Ltd. |
|
| TR01 | Transfer of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20190226 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |