CN107992491A - A kind of method and device of distributed file system, data access and data storage - Google Patents
A kind of method and device of distributed file system, data access and data storage Download PDFInfo
- Publication number
- CN107992491A CN107992491A CN201610949569.1A CN201610949569A CN107992491A CN 107992491 A CN107992491 A CN 107992491A CN 201610949569 A CN201610949569 A CN 201610949569A CN 107992491 A CN107992491 A CN 107992491A
- Authority
- CN
- China
- Prior art keywords
- client
- namenode
- data
- data access
- access request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
- G06F21/6227—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database where protection concerns the structure of data, e.g. records, types, queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
This application involves data processing field, more particularly to the method and device of a kind of distributed file system, data access and data storage, to solve the problems, such as that the framework of HDFS clusters in the prior art can cause whole HDFS not work normally in the case where NameNode breaks down；System provided by the embodiments of the present application includes：Upper strata namenode, multiple lower floor's namenodes and multiple back end；Upper strata namenode, for receiving the data access request of client transmission, lower floor corresponding with client namenode is forwarded to based on client with the mapping relations of lower floor namenode by data access request；Lower floor's namenode, for receiving the data access request forwarded by upper strata namenode, and the corresponding back end of searching data access request, client is back to by the information of the back end found；Back end, for being provided and the corresponding data message of data access request to client.
Description
Technical field
This application involves data processing field, more particularly to a kind of distributed file system, data access and data storage
Method and device.
Background technology
With the continuous development of network technology, we have welcome the big data epoch.Under the big data epoch, distribution is deposited
Storage, the read-write of big file all become much-talked-about topic.Distributed file system (Hadoop Distributed File System,
HDFS it is) existing for big data, HDFS is capable of providing the data access of high-throughput, is adapted to have ultra-large data
The application program of collection.
At present, HDFS clusters are by a namenode (NameNode) and thousands of a back end (DataNode) group
Into, wherein, NameNode is responsible for the NameSpace of file system (as opened, closing, again as a central server
Name file or catalogue) and access of the client to file, it also is responsible for determining mapping of the data block to specific DataNode.
DataNode it is upper really storage data block, be responsible for processing file system client read-write requests, while also can periodically to
NameNode sends heartbeat signal and bulk state reports that (when a DataNode starts, it can scan local file system, production
The list of the corresponding all HDFS data blocks of one these local file of life, is then sent to NameNode as report, this
Report is exactly bulk state report).Due to the reading and operation of all metadata informations in HDFS clusters be required for
NameNode communicates, this is resulted in when NameNode breaks down, and whole HDFS can not also be worked normally.In addition, work as
Cluster greatly to a certain extent after, memory that NameNode processes use can be very big, can also influence the normal use of HDFS.
To sum up, the framework of HDFS clusters can cause whole HDFS in the case where NameNode breaks down in the prior art
The problem of can not working normally.
The content of the invention
The embodiment of the present application provides the method and device of a kind of distributed file system, data access and data storage, uses
It can cause whole HDFS can not be just in the case where NameNode breaks down to solve the framework of HDFS clusters in the prior art
Often the problem of work.
The embodiment of the present application provides a kind of distributed file system, which includes：Upper strata namenode HNameNode,
Multiple lower floor's namenode NameNode and multiple back end DataNode；
The HNameNode, for receiving the data access request of client transmission, based on client and NameNode
The data access request is forwarded to NameNode corresponding with the client by mapping relations；
The NameNode, for receiving the data access request forwarded by the HNameNode, and searches the data
The corresponding DataNode of access request, the client is back to by the information of the DataNode found；
The DataNode, for being provided and the corresponding data message of the data access request to the client.
The embodiment of the present application provides a kind of data access method of distributed file system, including：
Upper strata namenode HNameNode receives the data access request that client is sent；
Based on the mapping relations of client and lower floor namenode NameNode by the data access request be forwarded to
The corresponding NameNode of the client.
The another embodiment of the application provides a kind of data access method of distributed file system, including：
Lower floor namenode NameNode receives the data from client forwarded by upper strata namenode HNameNode
Access request；
Based on the data access request received, back end corresponding with the data access request is searched
DataNode；
The information of the DataNode found is sent to client, so that the client is obtained to the DataNode
Data message.
The embodiment of the present application provides a kind of date storage method of distributed file system, including：
Upper strata namenode HNameNode receives the data storage request that client is sent；
The available situation information that resource is locally stored reported based on each back end DataNode, selection are used to store
The DataNode of data；
The information of the DataNode for storing data of selection is sent to the client, so that the client will
The data of request storage are write in the DataNode of selection.
The embodiment of the present application provides a kind of data access device of distributed file system, including：
Receiving module, for receiving the data access request of client transmission；
Sending module, for based on the mapping relations of client and lower floor namenode NameNode by the data access
Request is forwarded to NameNode corresponding with the client.
The another embodiment of the application provides a kind of data access device of distributed file system, including：
Receiving module, please by the data access from client of upper strata namenode HNameNode forwardings for receiving
Ask；
Searching module, for based on the data access request received, searching corresponding with the data access request
Back end DataNode；
Sending module, for sending the information of the DataNode found to client, so that the client is to institute
State DataNode and obtain data message.
The embodiment of the present application provides a kind of data storage device of distributed file system, including：
Receiving module, for receiving the data storage request of client transmission；
Selecting module, for the available situation that resource the is locally stored letter reported based on each back end DataNode
Breath, selection are used for the DataNode for storing data；
Sending module, for sending the information of the DataNode for storing data of selection to the client, with
Toilet states client and writes the data for asking storage in the DataNode of selection.
In the embodiment of the present application, two layers of NameNode frameworks are employed in HDFS clusters, HNameNode connects when upper strata
When receiving the data access request of client transmission, can the mapping relations based on client and lower floor NameNode by data access
Request is forwarded to lower floor NameNode corresponding with the client, then by definite lower floor NameNode find out its it is local with
The corresponding DataNode of the data access request simultaneously informs the information of the DataNode to client, so as to client to this
DataNode obtains corresponding data.In this way, NameNode single in the prior art can be effectively prevented from
The problem of causing whole HDFS not work normally during failure.In addition, in the embodiment of the present application, data access is being carried out
During, upper strata HNameNode can dock received data access request and carry out authentication and scope check, only have
The legal identity and data access request sent of client with associated rights can just be forwarded to lower floor NameNode, so that
Security protection has been carried out to the data in lower floor NameNode and the local DataNode of each NameNode.
Brief description of the drawings
Fig. 1 is the networking schematic diagram for the distributed file system that the embodiment of the present application one provides；
Fig. 2 is the data access method flow chart for the distributed file system that the embodiment of the present application two provides；
Fig. 3 is the date storage method flow chart for the distributed file system that the embodiment of the present application three provides；
Fig. 4 is the data access device structure chart for the distributed file system that the embodiment of the present application four provides；
Fig. 5 is the data access device structure chart for the distributed file system that the embodiment of the present application five provides；
Fig. 6 is the data storage device structure chart for the distributed file system that the embodiment of the present application six provides.
Embodiment
In the embodiment of the present application, two layers of NameNode frameworks are employed in HDFS clusters, HNameNode connects when upper strata
When receiving the data access request of client transmission, can the mapping relations based on client and lower floor NameNode by data access
Request is forwarded to lower floor NameNode corresponding with the client, then by definite lower floor NameNode find out its it is local with
The corresponding DataNode of the data access request simultaneously informs the information of the DataNode to client, so as to client to this
DataNode obtains corresponding data.In this way, NameNode single in the prior art can be effectively prevented from
The problem of causing whole HDFS not work normally during failure.In addition, in the embodiment of the present application, data access is being carried out
During, upper strata HNameNode can dock received data access request and carry out authentication and scope check, only have
The legal identity and data access request sent of client with associated rights can just be forwarded to lower floor NameNode, so that
Security protection has been carried out to the data in lower floor NameNode and the local DataNode of each NameNode.
As shown in Figure 1, the networking schematic diagram of the distributed file system provided for the embodiment of the present application one, the system bag
Include：Upper strata namenode (Hierarchical NameNode, referred to as HNameNode), multiple lower floor's namenodes are (referred to as
For NameNode) and multiple back end (DataNode).
Here, HNameNode as whole distributed file system (Hadoop Distributed File System,
HDFS chief coordinator person and global administration person).HNameNode can establish a connection with N number of NameNode respectively, work as client
When end needs to access the data in HDFS, first have to send data access request to HNameNode, here, outside HDFS
Client for can only see a HNameNode, i.e. HNameNode is external unique visible node in HDFS, can be with
Access or storage for clients providing data etc. service.
In specific implementation, after HNameNode receives the data access request of client transmission, client can be based on
Determine the data access request being forwarded to NameNode corresponding with client with the mapping relations of NameNode, by determining
The NameNode gone out carries out the processing of follow-up data access process.In addition, HNameNode can also be received by each
NameNode periodically reports the running state information of itself, and according to the running state information, adjusts local point of NameNode
The DataNode matched somebody with somebody.If there is a NameNode not report the operating status of itself on schedule in N number of NameNode of lower floor
Information or the running state information reported are shown as abnormal operational conditions, then HNameNode can identify this
NameNode is in improper working status, and local the managed each DataNode of the NameNode are distributed to it
In the region that its NameNode is administered.
Here, HNameNode can also directly be built with local the managed each DataNode of N number of NameNode respectively
Vertical connection relation, the available situation information of resource is locally stored so that each DataNode is periodically reported to HNameNode.When
When HNameNode receives the data storage request that client is sent, it can be locally stored based on what each DataNode was reported
The available situation information of resource, selection are used for the DataNode for storing data, and the data of client request storage are write selection
DataNode in.When selection is used to store the DataNode of data, can be deposited according to each DataNode locals reported
The size for storing up the free space of resource selects the DataNode of memory space abundance.
In addition, HNameNode can also be disposed by the way of active and standby, can be immediately when HNameNode breaks down
Start spare HNameNode to be responsible for coordinating and manage whole distributed file system.
Here, each NameNode in HDFS clusters is in addition to the HNameNode with upper strata maintains the connection, also
Local DataNode establishes connection relation with oneself.Here, DataNode local NameNode can have one,
Can also have multiple, N number of NameNode described in the networking schematic diagram shown in Fig. 1 is respectively established with a DataNode
Connection relation, only as one for example, in practical applications, NameNode is local can multiple DataNode, and
And NameNode can establish a connection with these local DataNode, regularly receive what each DataNode was reported
Running state information, when DataNode breaks down, to respond rapidly, by the DataNode to break down
Data block be reassigned to other normal DataNode.
In distributed file system provided by the embodiments of the present application, total associations of the upper strata HNameNode as whole HDFS
Tune person and global administration person, are responsible for the coordination and access of storage resource in whole HDFS clusters, while are responsible for monitoring lower floor
The function situation of NameNode, and NameNode is only responsible for the NameSpace of its local DataNode and monitors each
The function situation of DataNode, so that effectively solving when NameNode single in the prior art breaks down causes entirely
The problem of HDFS can not be worked normally.
As shown in Fig. 2, the data access method flow chart of the distributed file system provided for the embodiment of the present application two, bag
Include following steps：
S201：Client sends data access request to HNameNode, and authentication is carried in the data access request
Information.
Here, authentication information can be by other Verification Systems in Hadoop clusters or gateway offer to visitor
The identity authentication result at family end, if the identity authentication result shows that the client is legal, HNameNode also will be considered to this
Client is legal.
S202：HNameNode checks whether the authentication information carried in the data access request is legal, should if confirming
Authentication information is legal, then continues to execute S203；Otherwise, end is handled.
S203：HNameNode has the control of authority table of the access rights of client, the authority that accesses inspection based on record
Look into, if scope check status display is by state, continue to execute S204；Otherwise terminate to handle.
Here, the control of authority table that record has the access rights of client, which may be considered, to be configured or led in advance by administrator
Enter to HNameNode, the operation object and the type of data manipulation accessed in the control of authority table comprising request.Wherein,
Operation object can be title of user name, IP address or application software etc..The type of data manipulation can be directed to HDFS
The operation of middle catalogue, can also be the operation for file in HDFS.Wherein, the operation to catalogue in HDFS can include to mesh
The establishment of record, renaming, movement, deletion, describe and list file all under catalogue etc., in addition, to catalogue in HDFS
Operation can also be client is currently licensed into other clients to the operating right of the catalogue.To file in HDFS
Operation can include upload, download, renaming, movement, deletion and description of file etc., in addition, the behaviour to file in HDFS
It can also be and client is currently licensed into other clients to the operating right of this document.
In specific implementation, the data access request that HNameNode determines to receive first is with user name, or with IP
The title of address or application software accesses, specify that perform the data access request specific operation object it
Afterwards, then go to judge whether the operation object has the authority for performing data manipulation corresponding with the data access request.Such as
The data access request that HNameNode is received needs to ask to delete some catalogue, then HNameNode can be directed to the data access
Request carries out scope check, it is first determined its operation object is the title of user name, IP address or application software, however, it is determined that is gone out
It is that the data access request is that the access carried out in a manner of user name goes to judge whether the user's name deletes with execution afterwards again
Except the authority of some catalogue.If the determination result is YES, then the data access request has passed through above-mentioned scope check.
S204：The mapping relations of NameNodes of the HNameNode based on client and lower floor forward data access request
To NameNode corresponding with the client.
Here, the mapping relations of the NameNode of client and lower floor can be pre-stored in the HNameNode of upper strata
, it can also be and carry in the data access request that client is sent, got in real time by upper strata HNameNode.
S205：NameNode searches data section corresponding with data access request based on the data access request received
Point DataNode.
S206：NameNode sends the information of the DataNode found to client.
S207：Client obtains corresponding data message to the DataNode found by NameNode.
In the embodiment of the present application, during data access is carried out, upper strata HNameNode can dock received data and visit
Ask that request carries out authentication and scope check, the number that the client only with legal identity and with associated rights is sent
Lower floor NameNode can be just forwarded to according to access request, so as to event occur solving NameNode single in the prior art
On the basis of the problem of causing whole HDFS not work normally during barrier, also lower floor NameNode and each NameNode is had jurisdiction over
Data in area in DataNode have carried out security protection.
In force, if not using two layers of NameNode framework, directly configured in HDFS clusters multiple independent
NameNode, does not configure HnameNode, can also avoid Single Point of Faliure, but this mode needs each in cluster
DataNode periodically reports the available resources situation being locally stored to all NameNode, so on the one hand can cause network
The waste of resource, on the other hand, since what all NameNode can receive that each DataNode reports is locally stored
Available resources situation, in fact it could happen that at a time all NameNode think some DataNode storage resource abundance
And available situation, so that all NameNode indicate that client stores data to same DataNode, finally make
Into data memory error.To avoid the above problem, it is distributed in the application following embodiments based on two layers of NameNode framework
The data storage of formula file system, in the method, the HNameNode that DataNode need to be only responsible for into cluster reports this
The available situation information of ground storage resource, without reporting the available resources situation being locally stored to all NameNode, is referred to
The description of embodiment three.
As shown in figure 3, the date storage method flow chart of the distributed file system provided for the embodiment of the present application three, bag
Include following steps：
S301：Client sends data storage request to HNameNode.
S302：DataNode reports the available situation information that resource is locally stored to HNameNode.
Here, step S301 and step S302 is two separate steps, on execution sequence in no particular order.And
And DataNode can periodically report the available situation information that resource is locally stored to HNameNode.
S303：The available situation that resource the is locally stored letter that HNameNode is reported based on each back end DataNode
Breath, selection are used for the DataNode for storing data.
S304：HNameNode sends the information of the DataNode for storing data of selection to client.
S305：Client writes the data for asking storage in the DataNode of selection.
In the embodiment of the present application, all DataNode are only reported to upper strata HNameNode and resource are locally stored in HDFS
Available situation information, so during data storage is carried out, upper strata HNameNode can be based on each back end
Resource that what DataNode was reported be locally stored can select the DataNode for storing data with situation information.This profit
The mode of system memory resources allotment is carried out with upper strata HNameNode, is gone out in the NameNode for solving single in the prior art
On the basis of the problem of causing whole HDFS not work normally during existing failure, multiple NameNode are also effectively prevent to same
The resource of DataNode storage resources carries out contention the problem of seizing, causing data memory error.
Based on same application design, a kind of data access side of distributed file system is additionally provided in the embodiment of the present application
The data access device of the distributed file system of method binding, due to the principle that the device solves the problems, such as and the embodiment of the present application point
The data access method of cloth file system is similar, therefore the implementation of the device may refer to the implementation of method, repeats part not
Repeat again.
As shown in figure 4, the data access device structure chart of the distributed file system provided for the embodiment of the present application four, bag
Include following steps：
Receiving module 41, for receiving the data access request of client transmission；
Sending module 42, for being visited the data based on client and the mapping relations of lower floor namenode NameNode
Ask that request is forwarded to NameNode corresponding with the client.
Alternatively, authentication information is carried in the data access request；
The sending module 42 is specifically used for：
After the definite authentication information is legal, the data access request is forwarded to and the client pair
The NameNode answered.
Alternatively, described device further includes：
Scope check module 43, for having the control of authority table of the access rights of client, the power of accessing based on record
Limit checks；In access authoritychecking by afterwards, using the sending module by the data access request be forwarded to it is described
The corresponding NameNode of client.
Alternatively, the scope check module 43 is specifically used for：
There is the control of authority table of the access rights of client based on record, determine to perform the operation of the data access request
Object simultaneously judges whether the operation object has the authority for performing data manipulation corresponding with the data access request.
As shown in figure 5, the data access device structure chart of the distributed file system provided for the embodiment of the present application five, bag
Include following steps：
Receiving module 51, for receiving the data access from client forwarded by upper strata namenode HNameNode
Request；
Searching module 52, for based on the data access request received, searching and the data access request pair
The back end DataNode answered；
Sending module 53, for sending the information of the DataNode found to client, so as to the client to
The DataNode obtains data message.
Based on same application design, a kind of data storage side of distributed file system is additionally provided in the embodiment of the present application
The data storage device of the distributed file system of method binding, due to the principle that the device solves the problems, such as and the embodiment of the present application point
The date storage method of cloth file system is similar, therefore the implementation of the device may refer to the implementation of method, repeats part not
Repeat again.
As shown in fig. 6, the data storage device structure chart of the distributed file system provided for the embodiment of the present application six, bag
Include following steps：
Receiving module 61, for receiving the data storage request of client transmission；
Selecting module 62, for the available situation that resource the is locally stored letter reported based on each back end DataNode
Breath, selection are used for the DataNode for storing data；
Sending module 63, for sending the information of the DataNode for storing data of selection to the client,
So that the client writes the data for asking storage in the DataNode of selection.
It should be understood by those skilled in the art that, embodiments herein can be provided as method, system or computer program
Product.Therefore, the application can use the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware
Apply the form of example.Moreover, the application can use the computer for wherein including computer usable program code in one or more
The computer program production that usable storage medium is implemented on (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.)
The form of product.
The application is with reference to according to the method, apparatus (system) of the embodiment of the present application and the flow of computer program product
Figure and/or block diagram describe.It should be understood that it can be realized by computer program instructions every first-class in flowchart and/or the block diagram
The combination of flow and/or square frame in journey and/or square frame and flowchart and/or the block diagram.These computer programs can be provided
The processors of all-purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce
A raw machine so that the instruction performed by computer or the processor of other programmable data processing devices, which produces, to be used in fact
The device for the function of being specified in present one flow of flow chart or one square frame of multiple flows and/or block diagram or multiple square frames.
These computer program instructions, which may also be stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory, which produces, to be included referring to
Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one square frame of block diagram or
The function of being specified in multiple square frames.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that counted
Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, thus in computer or
The instruction performed on other programmable devices is provided and is used for realization in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in a square frame or multiple square frames.
Although having been described for the preferred embodiment of the application, those skilled in the art once know basic creation
Property concept, then can make these embodiments other change and modification.So appended claims be intended to be construed to include it is excellent
Select embodiment and fall into all change and modification of the application scope.
Obviously, those skilled in the art can carry out the application essence of the various modification and variations without departing from the application
God and scope.In this way, if these modifications and variations of the application belong to the scope of the application claim and its equivalent technologies
Within, then the application is also intended to comprising including these modification and variations.
Claims (18)
1. a kind of distributed file system, it is characterised in that the system includes：Upper strata namenode HNameNode, multiple lower floors
Namenode NameNode and multiple back end DataNode；
The HNameNode, for receiving the data access request of client transmission, the mapping based on client and NameNode
The data access request is forwarded to NameNode corresponding with the client by relation；
The NameNode, for receiving the data access request forwarded by the HNameNode, and searches the data access
Corresponding DataNode is asked, the information of the DataNode found is back to the client；
The DataNode, for being provided and the corresponding data message of the data access request to the client.
2. the system as claimed in claim 1, it is characterised in that the HNameNode is additionally operable to：
Receive the data storage request that client is sent；
The available situation information that resource is locally stored reported based on each DataNode, selection are used to store data
DataNode, and the information of the DataNode for storing data of selection is sent to the client.
3. the system as claimed in claim 1, it is characterised in that the NameNode is additionally operable to：
The running state information of itself is reported to the HNameNode；
The HNameNode is additionally operable to：The running state information reported according to the NameNode, adjustment NameNode are local
The DataNode of distribution.
4. the system as claimed in claim 1, it is characterised in that carry authentication information in the data access request；
The HNameNode is specifically used for：
After the definite authentication information is legal, the data access request is forwarded to corresponding with the client
NameNode。
5. system as claimed in claim 4, it is characterised in that the HNameNode is specifically used for：
After the definite authentication information is legal, there is the control of authority of the access rights of the client based on record
Table, access scope check；
In access authoritychecking by afterwards, the data access request being forwarded to corresponding with the client
NameNode。
6. system as claimed in claim 5, it is characterised in that the HNameNode is specifically used for being carried out according to following steps
Access authoritychecking：
Determine the operation object of the execution data access request, and judge whether the operation object has and perform and the data
The authority of the corresponding data manipulation of access request.
7. a kind of data access method of distributed file system, it is characterised in that this method includes：
Upper strata namenode HNameNode receives the data access request that client is sent；
Based on the mapping relations of client and lower floor namenode NameNode by the data access request be forwarded to it is described
The corresponding NameNode of client.
8. the method for claim 7, it is characterised in that carry authentication information in the data access request；
The data access request is forwarded to NameNode corresponding with the client by the HNameNode, including：
After the definite authentication information is legal, the data access request is forwarded to corresponding with the client
NameNode。
9. method as claimed in claim 8, it is characterised in that the HNameNode is determining the authentication information conjunction
After method, further include：
There is the control of authority table of the access rights of the client based on record, access scope check；
In access authoritychecking by afterwards, the data access request being forwarded to corresponding with the client
NameNode。
10. method as claimed in claim 9, it is characterised in that the access rights for having the client based on record
Control of authority table, access scope check, including：
There is the control of authority table of the access rights of the client based on record, determine to perform the operation of the data access request
Object simultaneously judges whether the operation object has the authority for performing data manipulation corresponding with the data access request.
11. a kind of data access method of distributed file system, it is characterised in that this method includes：
Lower floor namenode NameNode receives the data access from client forwarded by upper strata namenode HNameNode
Request；
Based on the data access request received, back end corresponding with the data access request is searched
DataNode；
The information of the DataNode found is sent to client, so that the client obtains data to the DataNode
Information.
12. a kind of date storage method of distributed file system, it is characterised in that this method includes：
Upper strata namenode HNameNode receives the data storage request that client is sent；
The available situation information that resource is locally stored reported based on each back end DataNode, selection are used to store data
DataNode；
The information of the DataNode for storing data of selection is sent to the client, so that the client will ask
The data of storage are write in the DataNode of selection.
13. a kind of data access device of distributed file system, it is characterised in that the device includes：
Receiving module, for receiving the data access request of client transmission；
Sending module, for based on the mapping relations of client and lower floor namenode NameNode by the data access request
It is forwarded to NameNode corresponding with the client.
14. device as claimed in claim 13, it is characterised in that authentication letter is carried in the data access request
Breath；
The sending module is specifically used for：
After the definite authentication information is legal, the data access request is forwarded to corresponding with the client
NameNode。
15. device as claimed in claim 14, it is characterised in that described device further includes：
Scope check module, for there is the control of authority table of the access rights of the client based on record, access authority
Check；
In access authoritychecking by afterwards, being forwarded to the data access request and the client using the sending module
Hold corresponding NameNode.
16. device as claimed in claim 15, it is characterised in that the scope check module is specifically used for：
There is the control of authority table of the access rights of the client based on record, determine to perform the behaviour of the data access request
Make object and judge whether the operation object has the authority for performing data manipulation corresponding with the data access request.
17. a kind of data access device of distributed file system, it is characterised in that the device includes：
Receiving module, for receiving the data access request from client forwarded by upper strata namenode HNameNode；
Searching module, for based on the data access request received, searching number corresponding with the data access request
According to node DataNode；
Sending module, for sending the information of the DataNode found to client, so that the client is to described
DataNode obtains data message.
18. a kind of data storage device of distributed file system, it is characterised in that the device includes：
Receiving module, for receiving the data storage request of client transmission；
Selecting module, for the available situation information that resource is locally stored reported based on each back end DataNode, choosing
Select the DataNode for storing data；
Sending module, for sending the information of the DataNode for storing data of selection to the client, with toilet
Client is stated to write the data for asking storage in the DataNode of selection.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610949569.1A CN107992491A (en) | 2016-10-26 | 2016-10-26 | A kind of method and device of distributed file system, data access and data storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610949569.1A CN107992491A (en) | 2016-10-26 | 2016-10-26 | A kind of method and device of distributed file system, data access and data storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107992491A true CN107992491A (en) | 2018-05-04 |
Family
ID=62029391
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610949569.1A Pending CN107992491A (en) | 2016-10-26 | 2016-10-26 | A kind of method and device of distributed file system, data access and data storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107992491A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109246190A (en) * | 2018-08-07 | 2019-01-18 | 深圳市先河系统技术有限公司 | network addressing method, data editing method, device and storage medium |
| CN109302497A (en) * | 2018-11-29 | 2019-02-01 | 北京京东尚科信息技术有限公司 | Data processing method, access agent device and system based on HADOOP |
| CN112528348A (en) * | 2021-01-13 | 2021-03-19 | 何春江 | Web information safe storage method |
| CN114201446A (en) * | 2021-11-22 | 2022-03-18 | 苏州浪潮智能科技有限公司 | Method and system for realizing HDFS (Hadoop distributed File System) remote storage mounting |
| CN114374706A (en) * | 2022-01-11 | 2022-04-19 | 北京易智时代数字科技有限公司 | Content security management method based on distributed architecture |
| CN116055493A (en) * | 2023-03-31 | 2023-05-02 | 北京比格大数据有限公司 | Cross-architecture data acquisition method, gateway node and cross-architecture platform |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101520805A (en) * | 2009-03-25 | 2009-09-02 | 中兴通讯股份有限公司 | Distributed file system and file processing method thereof |
| CN102546664A (en) * | 2012-02-27 | 2012-07-04 | 中国科学院计算技术研究所 | User and authority management method and system for distributed file system |
| US20130325812A1 (en) * | 2012-05-30 | 2013-12-05 | Spectra Logic Corporation | System and method for archive in a distributed file system |
| US20140067992A1 (en) * | 2012-08-28 | 2014-03-06 | Fujitsu Limited | Computer product, communication node, and transmission control method |
| CN103647797A (en) * | 2013-11-15 | 2014-03-19 | 北京邮电大学 | Distributed file system and data access method thereof |
| CN104023058A (en) * | 2014-06-03 | 2014-09-03 | 浙江宇视科技有限公司 | Method and system for managing and allocating storage resources |
-
2016
- 2016-10-26 CN CN201610949569.1A patent/CN107992491A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101520805A (en) * | 2009-03-25 | 2009-09-02 | 中兴通讯股份有限公司 | Distributed file system and file processing method thereof |
| CN102546664A (en) * | 2012-02-27 | 2012-07-04 | 中国科学院计算技术研究所 | User and authority management method and system for distributed file system |
| US20130325812A1 (en) * | 2012-05-30 | 2013-12-05 | Spectra Logic Corporation | System and method for archive in a distributed file system |
| US20140067992A1 (en) * | 2012-08-28 | 2014-03-06 | Fujitsu Limited | Computer product, communication node, and transmission control method |
| CN103647797A (en) * | 2013-11-15 | 2014-03-19 | 北京邮电大学 | Distributed file system and data access method thereof |
| CN104023058A (en) * | 2014-06-03 | 2014-09-03 | 浙江宇视科技有限公司 | Method and system for managing and allocating storage resources |
Non-Patent Citations (1)
| Title |
|---|
| 陈燕，张金松: "《大数据技术及其应用》", 31 December 2015 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109246190A (en) * | 2018-08-07 | 2019-01-18 | 深圳市先河系统技术有限公司 | network addressing method, data editing method, device and storage medium |
| CN109302497A (en) * | 2018-11-29 | 2019-02-01 | 北京京东尚科信息技术有限公司 | Data processing method, access agent device and system based on HADOOP |
| CN112528348A (en) * | 2021-01-13 | 2021-03-19 | 何春江 | Web information safe storage method |
| CN112528348B (en) * | 2021-01-13 | 2022-05-20 | 何春江 | Web information safe storage method |
| CN114201446A (en) * | 2021-11-22 | 2022-03-18 | 苏州浪潮智能科技有限公司 | Method and system for realizing HDFS (Hadoop distributed File System) remote storage mounting |
| CN114201446B (en) * | 2021-11-22 | 2024-01-23 | 苏州浪潮智能科技有限公司 | Method and system for realizing remote storage mounting of HDFS (Hadoop distributed File System) |
| CN114374706A (en) * | 2022-01-11 | 2022-04-19 | 北京易智时代数字科技有限公司 | Content security management method based on distributed architecture |
| CN114374706B (en) * | 2022-01-11 | 2024-05-28 | 北京易智时代数字科技有限公司 | Content security management method based on distributed architecture |
| CN116055493A (en) * | 2023-03-31 | 2023-05-02 | 北京比格大数据有限公司 | Cross-architecture data acquisition method, gateway node and cross-architecture platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11132227B2 (en) | Monitoring and automatic scaling of data volumes | |
| CN107992491A (en) | A kind of method and device of distributed file system, data access and data storage | |
| US9971823B2 (en) | Dynamic replica failure detection and healing | |
| CN107229520B (en) | Data center operating system | |
| US9825956B2 (en) | Systems and methods for access permission revocation and reinstatement | |
| TWI473029B (en) | Extensible and programmable multi-tenant service architecture | |
| KR102013004B1 (en) | Dynamic load balancing in a scalable environment | |
| US8108352B1 (en) | Data store replication for entity based partition | |
| CN103608798B (en) | Group document services | |
| CN113515499B (en) | Database service method and system | |
| CN113169952A (en) | Container cloud management system based on block chain technology | |
| JP7389793B2 (en) | Methods, devices, and systems for real-time checking of data consistency in distributed heterogeneous storage systems | |
| CN106790092B (en) | Remote procedure call server control system and method | |
| US11106497B2 (en) | Distributed scheduling in a virtual machine environment | |
| CN102640108A (en) | Monitoring of replicated data instances | |
| CN104598459A (en) | Database processing method and system and data access method and system | |
| US11922155B2 (en) | Post-upgrade debugging in a remote network management platform | |
| CN111698126B (en) | Information monitoring method, system and computer readable storage medium | |
| CN112364110A (en) | Metadata management method, device and equipment and computer storage medium | |
| CN103488526A (en) | System and method for locking business resource in distributed system | |
| US11582083B2 (en) | Multi-tenant event sourcing and audit logging in a cloud-based computing infrastructure | |
| US20130239004A1 (en) | System and method for providing an in-memory data grid application container | |
| US9813496B2 (en) | Method and apparatus for collaborating in a work environment | |
| US8788474B2 (en) | Inode event notification for cluster file systems | |
| CN107832159A (en) | Manage the method and computer-readable recording medium of distributed lock |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180504 |
|
| RJ01 | Rejection of invention patent application after publication |