CN108241548A - A kind of file reading based on distributed system - Google Patents
A kind of file reading based on distributed system Download PDFInfo
- Publication number
- CN108241548A CN108241548A CN201611205070.6A CN201611205070A CN108241548A CN 108241548 A CN108241548 A CN 108241548A CN 201611205070 A CN201611205070 A CN 201611205070A CN 108241548 A CN108241548 A CN 108241548A
- Authority
- CN
- China
- Prior art keywords
- data block
- file
- version number
- data
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to a kind of file readings based on distributed system, pass through the scheme in the present invention, when the data block that client needs are read is the partial data in data entry, can avoid reading entire data entry, effective lifting system performance inside file system.Meanwhile data check function is provided, it ensure that stability and correctness in data distribution formula processing procedure.Allow User Defined data verification mode simultaneously, the result data most verified at last returns to corresponding user.
Description
【Technical field】
This invention address that Data Access Security, by distributed file reading process, to read data
Block carries out distributed validation, and ensures data without missing and complete.
【Background technology】
Traditional, after file server receives the file read request of client transmission, from least N number of storage service
Read block in device, storage server judge the version of data block to file interface server returned data block and version number
It is number whether identical, when the version number of at least N number of data block is identical, determine that the entry is effective, and then cache the data block read,
And the data block that client needs is sent to client in the data block of caching.Wherein, client often sends a file
Read requests, storage server just need to read multiple data blocks and its version number by storage server, and storage server is read
Data pressure increases, and client end response is caused to be delayed, system performance is relatively low.Meanwhile this distributed file reading manner is not
Only cost is big, inefficiency, and safety and reliability cannot be guaranteed.Therefore, the application proposes following scheme.
【Invention content】
In order to solve the above problem of the prior art, the present invention proposes a kind of file based on distributed system and reads
Method.The technical solution adopted by the present invention is as follows：
A kind of file reading based on distributed system, the method are applied to file server, the file clothes
Device and client, the management server and storage server of being engaged in communicate, which is characterized in that including：
The file read request that the client is sent is received, the file read request carries offset address and reads file
Data volume；
According to the offset address and the reading file data amount, the client needs are read at least one is determined
Target file entry belonging to a file destination data block and at least one file destination data block, a file include
At least one target file entry, each target file entry include at least one data block；
When at least one file destination data block is the partial data in the file destination slitting, to the pipe
Manage server and send and obtain request for the target file entry version number, and receive management server in response to the request and
The target file entry version number of feedback；
Request is obtained to the storage server transmission data block of each storage each file destination data block, and
It receives each storage server and responds the data block and obtain the file destination data block that request fed back and its right
The data block version number answered；
When the target file entry version number and the corresponding data block version number phase of each file destination data block
Meanwhile each file destination data block is sent to the client.
Preferably, each storage server response data block that receives obtains the mesh that request is fed back
After marking file data blocks and its corresponding data block version number, further include：
When the corresponding data block version number of each file destination data block is not exactly the same, to described in each storage
The storage server that data block is verified in file destination slitting sends the acquisition request of verification data block；
When the corresponding data block version number of each verification data block is identical, pass through data check erasure codes EC algorithms
Each verification data block is handled, obtains each file destination data block；
Each target data block is sent to the client.
Beneficial effects of the present invention include：When the data block that client needs are read is the partial data in data entry
When, it can avoid reading entire data entry, effective lifting system performance inside file system.Meanwhile provide data check work(
Can, it ensure that the stability and correctness in data distribution formula processing procedure.Allow User Defined data verification mode simultaneously,
The result data most verified at last returns to corresponding user.
【Description of the drawings】
Attached drawing described herein is to be used to provide further understanding of the present invention, and forms the part of the application, but
It does not constitute improper limitations of the present invention；
Fig. 1 is the flow chart of the method for the present invention.
【Specific embodiment】
Come that the present invention will be described in detail below in conjunction with attached drawing and specific embodiment, illustrative examples therein and say
It is bright to be only used for explaining the present invention, but not as a limitation of the invention.
Referring to attached drawing 1, a kind of file reading based on distributed system, the method is applied to file server,
The file server and client, management server and storage server communication, which is characterized in that including：
The file read request that the client is sent is received, the file read request carries offset address and reads file
Data volume.
According to the offset address and the reading file data amount, the client needs are read at least one is determined
Target file entry belonging to a file destination data block and at least one file destination data block, a file include
At least one target file entry, each target file entry include at least one data block.
When at least one file destination data block is the partial data in the file destination slitting, to the pipe
Manage server and send and obtain request for the target file entry version number, and receive management server in response to the request and
The target file entry version number of feedback.
Request is obtained to the storage server transmission data block of each storage each file destination data block, and
It receives each storage server and responds the data block and obtain the file destination data block that request fed back and its right
The data block version number answered.
When the target file entry version number and the corresponding data block version number phase of each file destination data block
Meanwhile each file destination data block is sent to the client.
Preferably, each storage server response data block that receives obtains the mesh that request is fed back
After marking file data blocks and its corresponding data block version number, further include：
When the corresponding data block version number of each file destination data block is not exactly the same, to described in each storage
The storage server that data block is verified in file destination slitting sends the acquisition request of verification data block；
When the corresponding data block version number of each verification data block is identical, pass through data check erasure codes EC algorithms
Each verification data block is handled, obtains each file destination data block；
Each target data block is sent to the client.
Preferably, each verification data block is handled by EC algorithms, obtains each number in target data slitting
According to block.
Preferably, allow User Defined data verification mode, the result data most verified at last returns to corresponding use
Family.
The above is only the better embodiment of the present invention, therefore all constructions according to described in present patent application range,
The equivalent change or modification that feature and principle are done, is included in the range of present patent application.
Claims (2)
1. a kind of file reading based on distributed system, the method is applied to file server, the file service
Device and client, management server and storage server communication, which is characterized in that including：
The file read request that the client is sent is received, the file read request carries offset address and reads file data
Amount；
According to the offset address and the reading file data amount, at least one mesh that the client needs are read is determined
Target file entry belonging to mark file data blocks and at least one file destination data block, a file are included at least
One target file entry, each target file entry include at least one data block；
When at least one file destination data block is the partial data in the file destination slitting, taken to the management
Device transmission be engaged in for target file entry version number acquisition request, and receives management server and is fed back in response to the request
Target file entry version number；
Request is obtained, and receive to the storage server transmission data block of each storage each file destination data block
Each storage server responds the data block and obtains the file destination data block that request fed back and its corresponding
Data block version number；
When data block version number corresponding with each file destination data block of the target file entry version number is identical,
Each file destination data block is sent to the client.
2. according to the method described in claim 2, it is characterized in that, described receive each storage server response number
After obtaining the file destination data block for asking to be fed back and its corresponding data block version number according to block, further include：
When the corresponding data block version number of each file destination data block is not exactly the same, to each storage target
The storage server that data block is verified in file slitting sends the acquisition request of verification data block；
When the corresponding data block version number of each verification data block is identical, by data check erasure codes EC algorithms to each
A verification data block is handled, and obtains each file destination data block；
Each target data block is sent to the client.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611205070.6A CN108241548A (en) | 2016-12-23 | 2016-12-23 | A kind of file reading based on distributed system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611205070.6A CN108241548A (en) | 2016-12-23 | 2016-12-23 | A kind of file reading based on distributed system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108241548A true CN108241548A (en) | 2018-07-03 |
Family
ID=62704157
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611205070.6A Pending CN108241548A (en) | 2016-12-23 | 2016-12-23 | A kind of file reading based on distributed system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108241548A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109558086A (en) * | 2018-12-03 | 2019-04-02 | 浪潮电子信息产业股份有限公司 | A kind of method for reading data, system and associated component |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070192542A1 (en) * | 2006-02-16 | 2007-08-16 | Svend Frolund | Method of operating distributed storage system |
| CN101334797A (en) * | 2008-08-04 | 2008-12-31 | 中兴通讯股份有限公司 | Distributed file systems and its data block consistency managing method |
| CN103547992A (en) * | 2011-03-31 | 2014-01-29 | 伊姆西公司 | System and method for maintaining consistent points in file systems using a prime dependency list |
| CN104272274A (en) * | 2013-12-31 | 2015-01-07 | 华为技术有限公司 | Data processing method and device in distributed file storage system |
| CN105426483A (en) * | 2015-11-19 | 2016-03-23 | 华为技术有限公司 | File reading method and device based on distributed system |
| CN106233264A (en) * | 2014-03-31 | 2016-12-14 | 亚马逊科技公司 | Use the file storage device of variable stripe size |
-
2016
- 2016-12-23 CN CN201611205070.6A patent/CN108241548A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070192542A1 (en) * | 2006-02-16 | 2007-08-16 | Svend Frolund | Method of operating distributed storage system |
| CN101334797A (en) * | 2008-08-04 | 2008-12-31 | 中兴通讯股份有限公司 | Distributed file systems and its data block consistency managing method |
| CN103547992A (en) * | 2011-03-31 | 2014-01-29 | 伊姆西公司 | System and method for maintaining consistent points in file systems using a prime dependency list |
| CN104272274A (en) * | 2013-12-31 | 2015-01-07 | 华为技术有限公司 | Data processing method and device in distributed file storage system |
| CN106233264A (en) * | 2014-03-31 | 2016-12-14 | 亚马逊科技公司 | Use the file storage device of variable stripe size |
| CN105426483A (en) * | 2015-11-19 | 2016-03-23 | 华为技术有限公司 | File reading method and device based on distributed system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109558086A (en) * | 2018-12-03 | 2019-04-02 | 浪潮电子信息产业股份有限公司 | A kind of method for reading data, system and associated component |
| CN109558086B (en) * | 2018-12-03 | 2019-10-18 | 浪潮电子信息产业股份有限公司 | A kind of method for reading data, system and associated component |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2781018A1 (en) | Systems, methods, and apparatus to monitor mobile internet activity | |
| MY201796A (en) | Authentication method and blockchain-based authentication data processing method and device | |
| WO2011043903A3 (en) | Network access control | |
| RU2017124135A (en) | INFORMATION PROCESSING DEVICE, MANAGEMENT METHOD FOR INFORMATION PROCESSING DEVICE, INFORMATION PROCESSING SYSTEM AND COMPUTER PROGRAM | |
| JP2015536603A5 (en) | ||
| JP2014053043A5 (en) | ||
| CN104008158A (en) | Data verification method, verification device and mobile terminal | |
| BRPI0823129B8 (en) | method and system for profiling user communication traffic in a data network, and, data network | |
| WO2012092602A3 (en) | Methods and systems for caching data communications over computer networks | |
| EP2698967A8 (en) | Social network data mining method for terminal user, and relevant method, device and system | |
| JP2009266202A5 (en) | Information processing apparatus, information processing apparatus control method, session management system, and program | |
| JP2014138324A5 (en) | ||
| RU2016136668A (en) | Caching Encrypted Content | |
| WO2014022555A3 (en) | Image repository systems and methods | |
| CN104348924A (en) | Method, system and device for domain name resolution | |
| EP2779669A3 (en) | Method and system for acquiring access rights to conditional access content | |
| CN103581040A (en) | Server distributing device and server distributing method | |
| CN104811394A (en) | Method and system for saving traffic for accessing server | |
| JP2016511987A5 (en) | ||
| CN104683459A (en) | Method for transmitting service data, and storage system | |
| CN106790334A (en) | A kind of page data transmission method and system | |
| EP2725514A3 (en) | Security information sharing system and execution method thereof | |
| CN104320447A (en) | Medium and small-sized cloud stored data synchronization method, compute readable medium and system | |
| CN108241548A (en) | A kind of file reading based on distributed system | |
| JP2013218505A5 (en) |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| CB02 | Change of applicant information |
Address after: 101399 No. 2 East Airport Road, Shunyi Airport Economic Core Area, Beijing (1st, 5th and 7th floors of Industrial Park 1A-4) Applicant after: Zhongke Star Map Co.,Ltd. Address before: 101399 Building 1A-4, National Geographic Information Technology Industrial Park, Guomen Business District, Shunyi District, Beijing Applicant before: GEOVIS TECHNOLOGY (BEIJING) Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180703 |
|
| RJ01 | Rejection of invention patent application after publication |