CN105511805B - The data processing method and device of cluster file system - Google Patents
The data processing method and device of cluster file system Download PDFInfo
- Publication number
- CN105511805B CN105511805B CN201510847053.1A CN201510847053A CN105511805B CN 105511805 B CN105511805 B CN 105511805B CN 201510847053 A CN201510847053 A CN 201510847053A CN 105511805 B CN105511805 B CN 105511805B
- Authority
- CN
- China
- Prior art keywords
- node
- memory node
- data
- memory
- read
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
Abstract
The invention discloses a kind of data processing equipments of cluster file system, the data processing equipment of the cluster file system includes: data processing module, for when memory node receives file read or write request, the data in memory block are read according to the solicited message in the read requests and the buffer area, or data are written into memory block according to the solicited message in said write request and the buffer area；Memory module, for the solicited message in the read or write request to be stored in buffer area；Synchronization module, the slave memory node for being synchronized to the solicited message in cluster file system.The present invention also proposes a kind of data processing method of cluster file system.In the present invention after primary storage node failure, it can guarantee the stability of cluster file system service according to the service request information in its buffer area, the reading or write-in process of unremitting adapter tube client from memory node.
Description
Technical field
The present invention relates to technical field of data processing more particularly to the data processing methods and dress of a kind of cluster file system
It sets.
Background technique
In recent years, cluster file system has become the new trend of technical development of computer.Cluster file system pass through by
Multiple single independent hosts organically combine and are connected into a systematic entirety, and rely on storage area network, externally mention
For the shared storage application of a large capacity file system, such as CIFS.Realize more concurrent users behaviour in storage area network
Make and big data is transmitted.
Even so, a possibility that increase of clustered node also increases node delay machine or service fault simultaneously.For example, working as
Client computer is when reading and writing a larger file into cluster, if accepting the service node failure of the read-write requests, even if service
It can be switched to other nodes in time, since there is no data buffer storages in the memory of other nodes, then the read-write operation of client computer
Or it can interrupt.
Summary of the invention
The main purpose of the present invention is to provide a kind of data processing method and device of cluster file system, it is intended to solve
Due to cluster service node failure, the technical issues of making read-write on client side operation disruption.
To achieve the above object, the data processing equipment of a kind of cluster file system provided by the invention, the cluster text
The data processing equipment of part system includes:
Memory module, for receiving file read or write request in memory node, and where the memory module
When memory node is main memory node, the solicited message in the read or write request is stored in its buffer area；
Synchronization module, the slave memory node for being synchronized to the solicited message in cluster file system；
Data processing module, for reading its storage according to the solicited message in the read requests and the buffer area
Data in area, or number is written into its memory block according to the solicited message in said write request and the buffer area
According to.
Optionally, the data processing equipment of the cluster file system includes:
Detection module when for receiving service request, detects the type of the service request；
Judgment module, for judging whether the service request is read or write request；
Respond module responds the service request if not being read or write request for the service request.
Optionally, the data processing equipment of the cluster file system further include:
Determining module, for when receiving file read or write request, determining that the read or write request is corresponding
Data whether need to verify；
Module is obtained, for when the corresponding data of the read or write request need to verify, obtaining user's input
Verification information；
The data processing module is also used to when the verification information is matched with the verification information prestored, according to described
Solicited message in read requests and the buffer area reads the data in its memory block, or according to said write request with
And data are written into its memory block in the solicited message in the buffer area.
Optionally, the data processing equipment of the cluster file system further include:
The detection module is also used to detect link connection state and operating status；
Switching module, for when detecting link connection failure or operation troubles, in normal operating conditions from
In memory node choose primary storage node, by described in selection from the address mark of memory node be main memory node address；
Update module, for by label the primary storage node address be sent to described in control node and selection from
Memory node, wherein the control node updates the primary storage saved using the primary storage node address received
Node address, and working condition is switched to main memory when receiving the primary storage node address by the memory node chosen
Store up node state.
Optionally, the detection module includes:
Link detecting unit sends the first detection data packet to the control node for timing and receives the control
The second response data packet that node is fed back based on the detection data packet；
Read and write detection unit, when for receiving the response data packet within a preset time interval, judge read and
It whether normal is written；
Judging unit, for determining normal operation when reading and being written normal, when reading and being written abnormal,
When determining operation troubles, and not receiving the response data packet within a preset time interval, the link connection event is determined
Barrier.
The present invention further provides a kind of data processing method of cluster file system, the data of the cluster file system
Processing method includes:
File read or write request is received in memory node, and when the memory node is main memory node, it is described
Memory node reads the data in its memory block, Huo Zhegen according to the solicited message in the read requests and the buffer area
Data are written into its memory block according to the solicited message in said write request and the buffer area；
Solicited message in the read or write request is stored in its buffer area by the memory node；
The solicited message is synchronized to the slave memory node in cluster file system by the memory node.
Optionally, the step of solicited message in the read or write request is stored in its buffer area by the memory node
Before further include:
When the memory node receives service request, the type of the service request is detected；
Judge whether the service request is read or write request；
If the service request be read or write request, execute it is described will it is described reading or write service request in
Solicited message is stored in the step of its buffer area；
It is described judge the step of whether service request is read or write request after, the cluster file system
Data processing method further include:
If the service request is not read or write request, the service request is responded.
Optionally, the memory node reads it according to the solicited message in the read requests and the buffer area and deposits
Data in storage area, or number is written into its memory block according to the solicited message in said write request and the buffer area
According to the step of before, the data processing method of the cluster file system further comprises the steps of:
The memory node determines whether the corresponding data of the read or write request need to verify；
When the corresponding data of the read or write request need to verify, the memory node obtains testing for user's input
Demonstrate,prove information；
When the verification information is matched with the verification information prestored, the memory node is executed according to the read requests
And the solicited message in the buffer area reads the data in its memory block, or according to said write request and described slow
Deposit the step of data are written into its memory block for the solicited message in area.
Optionally, the data processing method of the cluster file system further include:
When the memory node is main memory node, the memory node detects its link connection state and the storage
The operating status of node；
When the acquisition memory node real-time detection is to its link connection failure or the memory node operation troubles, institute
It states memory node and chooses primary storage node in the slave memory node in normal operating conditions, will be saved described in selection from storage
The address mark of point is main memory node address；
The host node address of label is sent to described in control node and selection from memory node, wherein institute
It states control node and updates the host node address saved, and the storage chosen using the host node address received
When node receives the host node address, working condition is switched to primary storage node state.
Optionally, the memory node detects the step of operating status of its link connection state and memory node packet
It includes:
The memory node timing sends the first detection data packet to the control node；
Receive the second response data packet that the control node is fed back based on the detection data packet:
When receiving the response data packet within a preset time interval, the reading and write-in of the memory node are judged
It is whether normal；
In the reading and normal write-in of the memory node, determines the memory node normal operation, deposited described
When storing up the reading and abnormal write-in of node, the memory node operation troubles is determined；
When not receiving the response data packet within a preset time interval, the link connection event of the memory node is determined
Barrier.
The data processing method and device of cluster file system proposed by the present invention receive file in memory node and read
Or write request, and when the memory node is main memory node, the memory node is according to read requests and described
Solicited message in buffer area reads the data in its memory block, or according in said write request and the buffer area
Data are written into its memory block for solicited message, and the solicited message in the read or write request is stored in by the memory node
Its buffer area, the solicited message is synchronized to the slave memory node in cluster file system by the memory node, in primary storage
After node failure, from memory node can according to the reading of the unremitting adapter tube client of service request information in its buffer area or
Write-in process guarantees the stability of cluster file system service.
Detailed description of the invention
Fig. 1 is the functional block diagram of the data processing equipment first embodiment of cluster file system of the present invention；
Functional module when Fig. 2 is other services outside the data processing equipment processing read-write of cluster file system of the present invention is shown
It is intended to；
Fig. 3 is the functional block diagram of the data processing equipment second embodiment of cluster file system of the present invention；
Fig. 4 is the refinement the functional block diagram of detection module in Fig. 3；
Fig. 5 is the functional block diagram of the data processing equipment 3rd embodiment of cluster file system of the present invention；
Fig. 6 is the flow diagram of the data processing method first embodiment of cluster file system of the present invention；
Fig. 7 is that the process when data processing method of cluster file system of the present invention handles other services outside reading and writing data is shown
It is intended to；
Fig. 8 is the flow diagram of the data processing method second embodiment of cluster file system of the present invention；
Fig. 9 is the refinement flow diagram of primary storage nodal test link connection state and operating status in Fig. 8；
Figure 10 is the flow diagram of the data processing method 3rd embodiment of cluster file system of the present invention.
The object of the invention is realized, the embodiments will be further described with reference to the accompanying drawings for functional characteristics and advantage.
Specific embodiment
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
The present invention provides a kind of data processing equipment of cluster file system.
Referring to Fig.1, Fig. 1 is the functional module signal of the data processing equipment first embodiment of cluster file system of the present invention
Figure.
It is emphasized that it will be apparent to those skilled in the art that functional block diagram shown in Fig. 1 is only one preferably real
Apply the exemplary diagram of example, function mould of the those skilled in the art around the data processing equipment of cluster file system shown in FIG. 1
Block can carry out the supplement of new functional module easily；The title of each functional module is self-defined title, is only used for auxiliary and understands ring
Each program function block of border monitoring device is not used in restriction technical solution of the present invention, and the core of technical solution of the present invention is,
The function to be reached of the functional module of each self-defined title.
In the present embodiment, the data processing equipment of the cluster file system includes:
Data processing module 10, for when memory node receives file read or write request, according to the reading
Request and the buffer area in solicited message read memory block in data, or according to said write request and it is described
Data are written into memory block for solicited message in buffer area；
Memory module 20, for the solicited message in the read or write request to be stored in its buffer area；
Memory module 20 can obtain the configuration parameter in control node in initial start-up, initialize, each
A memory node creates the buffer area, it is to be understood that the configuration parameter can be set by the user himself, and can also service
By producer's typing when device manufactures, and the configuration parameter can modify at any time, and be forwarded to by control node all
Memory node.
In the present embodiment, the configuration parameter includes:
Workgroup name, for defining the workgroup name of cluster file system；
Server name, for defining the title of each memory node of cluster file system；
Maximum number of connections, for defining the number of clients at most allowing while accessing cluster file system；
Unified store path can be accessed for defining the root of cluster file system by any node address
To same directory address or file；
Redundant network interfaces, each memory node for defining cluster file system are used as the network interface of redundancy, i.e.,
Network interface card；
Redundant network address, each memory node for defining cluster file system are used as the network of redundant network interfaces
Address (i.e. the corresponding IP of the network interface card), can be set multiple；
Redundant host address provides the public network address of access for defining cluster file system curstomer-oriented end, only
Settable one；
Data buffer area size, for defining on each memory node of cluster file system for storing in data cached
Deposit size.
Such as the memory space of the buffer area occupancy 20GB of memory node can be set in user, what the buffer area occupied deposits
Storage space must not exceed the memory maximum memory space of memory node server.
Synchronization module 30, the slave memory node for being synchronized to the solicited message in cluster file system
In the present embodiment, control node, can be to the service for being arranged to primary storage node in system start-up initialisation
Device sends message, controls the primary storage node startup program, and other memory nodes not will start service routine, can only receive
And the solicited message of the synchronous read or write request of primary storage node is stored, only primary storage node is receiving reading data
Or when write request, the read or write request is synchronized to it is all from memory node in cluster file system, from storage
The solicited message of the read or write request received is only stored in its buffer area by node, and without other any behaviour
Make.
After primary storage node receives reading data or the write request of client, the request of the request can be believed
Breath: it such as the I/O offset of process number, the process that client data reads or is written, is synchronized to all from memory node.
Referring to Fig. 2, memory node can determine the type of service request, be asked according to server when receiving service request
The type asked performs corresponding processing, i.e., the data processing equipment of the described cluster file system further include:
Detection module 40 when for receiving service request, detecting the type of the service request, judging the service request
It whether is read or write request；
Respond module 50 responds the service request if not being read or write request for the service request；
The data processing module 10, if being also used to the service request is read or write request, according to the reading
The solicited message in request and the buffer area is taken to read the data in memory block, or according to said write request and institute
Data are written into memory block for the solicited message stated in buffer area.
After primary storage node receives service request, detection module 40 can detect the service type of the request call,
In the request call reading data or write service, the solicited message of the request is sent to main memory by 30 ability of synchronization module
Node network interfaces are stored up, then by the network interface of primary storage node to be sent to the node address got from control node corresponding
It is stored in its buffer area from memory node, and by synchronization request will be received from the memory module 10 of memory node.
It is understood that primary storage node can not only receive reading data or write request, it is also possible to can connect
Other kinds of service request is received, modifies and requests such as configuration parameter, it is straight to respond the service request for respond module 50 at this time
The response request is connect, such as the corresponding parameter of the request is modified accordingly, but the memory module 20 will not will service
Request store to its buffer area, and synchronization module 20 service request will not be synchronized to it is all from memory node.
It is understood that in the present embodiment, due to system upgrade or maintenance etc., the primary storage node may
When can not continue as clients providing data reading or write service, primary storage node can select one to be in normal operating conditions
From memory node, the slave memory node being selected is switched to primary storage node, the reading data of adapter tube client or be written into
Journey.
The data processing equipment for the cluster file system that the present embodiment proposes, primary storage node are asked receiving data manipulation
It is any to be in normal work when needing to switch memory node by the synchronizing information of the request to all from memory node after asking
Make the slave memory node of state can adapter tube client data manipulation process, provide unremitting data manipulation clothes for client
Business, improves the availability of cluster file system.
Further, when primary storage node provides service for client, it is possible to break down, then primary storage node needs
Timing detects its link connection state and service running state of programs, to find failure in time, and makes respective handling, then base
The data processing equipment second embodiment of cluster file system of the present invention is proposed in first embodiment, referring to Fig. 3, the cluster text
The data processing equipment of part system further include:
The detection module 40, is also used to detect its link connection state and operating status；
Switching module 60, when for detecting link connection failure or operation troubles, in normal operating conditions from
In memory node choose primary storage node, by described in selection from the address mark of memory node be main memory node address；
Update module 70, for the host node address of label to be sent to described in control node and selection from depositing
Store up node, wherein the control node updates the host node address saved using the host node address received, and
When the memory node chosen receives the host node address, working condition is switched to primary storage node state.
When needing to switch memory node, switching module 60 can choose a slave memory node for being in normal operating conditions,
It is new primary storage address of node by the address mark of the slave service node being selected, update module 70 sends out its address
It send to control node, control node will be forwarded to client request later new primary storage node, described to be selected
It can be switched to primary storage node working condition from memory node, new primary storage node can read the reading of the data in its buffer area
Progress information is taken or be written, corresponding process is directly taken over, the data manipulation process interrupt of client will not be made.
It is understood that may be roughly the same from the performance of memory node, it is also possible to which performance difference is larger, so in master
When memory node switches over, if all very nearly the same from memory node performance, switching module 60 can be randomly selected at one
In normal operating conditions slave memory node as primary storage node, can if all performance differences from memory node are larger
Select a higher node of performance in normal operating conditions as new master according to from the performance sequence of memory node
Memory node.
The data processing equipment for the cluster file system that the present embodiment proposes, timing detect the link connection of primary storage node
State and service running state of programs, find the failure of primary storage node, and then switch primary storage node in time, mention for client
For unremitting reading and writing data service, it is ensured that the stability of cluster file system data processing service.
Referring to Fig. 4, the detection module 40 includes:
Link detecting unit 41 sends the first detection data packet to the control node for timing；
Receive the second response data packet that the control node is fed back based on the detection data packet:
Read and write detection unit 42, when for receiving the response data packet within a preset time interval, judge read with
And whether write-in is normal；
Judging unit 43, it is abnormal reading and being written for determining normal operation when reading and being written normal
When, when determining operation troubles, and not receiving the response data packet within a preset time interval, determine link connection event
Barrier.
In the present embodiment, it detects that the primary storage node breaks down in detection module 40, client can not be continued as
When end provides reading data or write service, primary storage node can select a slave memory node for being in normal operating conditions, will
The slave memory node being selected is switched to primary storage node, the reading data or write-in process of adapter tube client.
Further, it is the safety for improving cluster file system, is also proposed based on the first or second embodiments present invention
The 3rd embodiment of the data processing equipment of cluster file system, referring to Fig. 5, the data processing equipment of the cluster file system
Further include:
Determining module 80, for determining the read or write request pair when receiving file read or write request
Whether the data answered, which need, is verified；
In the present embodiment, when primary storage node receives file read or write request, the requested visit can be detected
The access level for the file asked, if the file is shared file, it can when being accessed by all users, then do not need to user
Identity authority verified, if the file is privately owned file, i.e., the described file can only be had the user of permissions
Access, then need to verify the identity authority of user, determine whether the file can be issued the client of access request
Access.
Module 90 is obtained, for when the corresponding data of the read or write request need to verify, obtaining user's input
Verification information；
It is privately owned file in the file of client request access, that is, when needing to verify, primary storage node needs to obtain user
The information of input verifies the permission of user.At this point, user may carry out register, primary storage node only needs
The information of control node forwarding is traversed, data of the crawl with username and password keyword or handle, and then obtained
The information of user's input, verifies the permission of user；If user not yet logs at this time, primary storage node can be to control node
Message is sent, notice control node sends prompting to client or control client pops up login interface, tests in user's input
After demonstrate,proving information, the verification information that the user inputs is forwarded to primary storage node by control node and is verified.
The data processing module 10 is also used to when the verification information is matched with the verification information prestored, according to institute
It states the solicited message in read requests and the buffer area and reads data in its memory block, or requested according to said write
And data are written into its memory block in the solicited message in the buffer area.
The data processing equipment for the cluster file system that the present embodiment proposes, after receiving data operation request,
The operating right of the data can be obtained, if the data need Authority Verification, obtain user input verification information, with
When the verification information of family input is matched with the verification information prestored, corresponding data manipulation is just executed, improves cluster file system
The safety of data.
The present invention further provides a kind of data processing methods of cluster file system.
Referring to Fig. 6, the data processing method of the cluster file system includes:
Step S10, when memory node receives file read or write request, the memory node is according to the reading
Solicited message in request and the buffer area reads the data in its memory block, or according to said write request and institute
Data are written into its memory block in the solicited message stated in buffer area；
Solicited message in the read or write request is stored in its buffer area by step S20, the memory node；
Memory node can obtain the configuration parameter in control node in initial start-up, initialize, storing
Node creates the buffer area, it is to be understood that the configuration parameter can be set by the user himself, and can also manufacture in server
When by producer's typing.
In the present embodiment, the configuration parameter includes:
Workgroup name, for defining the workgroup name of cluster file system；
Server name, for defining the title of each memory node of cluster file system；
Maximum number of connections, for defining the number of clients at most allowing while accessing cluster file system；
Unified store path can be accessed for defining the root of cluster file system by any node address
To same directory address or file；
Redundant network interfaces, each memory node for defining cluster file system are used as the network interface of redundancy, i.e.,
Network interface card；
Redundant network address, each memory node for defining cluster file system are used as the network of redundant network interfaces
Address (i.e. the corresponding IP of the network interface card), can be set multiple；
Redundant host address provides the public network address of access for defining cluster file system curstomer-oriented end, only
Settable one；
Data buffer area size, for defining on each memory node of cluster file system for storing in data cached
Deposit size.
Such as the memory space of the buffer area occupancy 20GB of memory node can be set in user, what the buffer area occupied deposits
Storage space must not exceed the memory maximum memory space of memory node server.
In the present embodiment, control node, can be to the service for being arranged to primary storage node in system start-up initialisation
Device sends message, controls the primary storage node startup program, and other memory nodes not will start service routine, can only receive
And the solicited message of the synchronous read or write request of primary storage node is stored, only primary storage node is receiving reading data
Or when write request, the read or write request is synchronized to it is all from memory node in cluster file system, from storage
The solicited message of the read or write request received is only stored in its buffer area by node, and without other any behaviour
Make.
After primary storage node receives reading data or the write request of client, the request of the request can be believed
Breath: it such as the I/O offset of process number, the process that client data reads or is written, is synchronized to all from memory node.
The solicited message is synchronized to the slave memory node in cluster file system by step S30, the memory node.
Primary storage node can respond the request after receiving reading data or write request, be believed according to the request
Breath, for clients providing data reading or write service.
It is understood that primary storage node can not only receive reading data or write request, it is also possible to can connect
Other kinds of service request is received, modifies and requests such as configuration parameter, at this time directly in response to the request, such as to the request pair
The parameter answered is modified accordingly, but not service request is stored to its buffer area, and will not be synchronous by service request
It is extremely all from memory node.
Referring to Fig. 7, memory node can determine the type of service request, be asked according to server when receiving service request
The type asked performs corresponding processing, i.e., the data processing method of the described cluster file system further include:
Step S40 detects the type of the service request when the memory node receives service request；
Step S50 judges whether the service request is read or write request；
If the service request is read or write request, execute the memory node according to the read requests and
Solicited message in the buffer area reads the data in its memory block, or according to said write request and the buffer area
In solicited message the step of data are written into its memory block.
After the step S50, the data processing method of the cluster file system further include:
Step S60 responds the service request if the service request is not read or write request.
After primary storage node receives service request, the service type of the request call can be detected, in the request
It calls reading data or when write service, the solicited message of the request is just sent to primary storage node network interfaces, then by
It is corresponding from memory node that the network interface of primary storage node is sent to the node address got from control node, and by from depositing
Storage node will receive synchronization request and be stored in its buffer area.
It is understood that in the present embodiment, due to system upgrade or maintenance etc., the primary storage node may
When can not continue as clients providing data reading or write service, primary storage node can select one to be in normal operating conditions
From memory node, the slave memory node being selected is switched to primary storage node, the reading data of adapter tube client or be written into
Journey.
The data processing method for the cluster file system that the present embodiment proposes, primary storage node are asked receiving data manipulation
It is any to be in normal work when needing to switch memory node by the synchronizing information of the request to all from memory node after asking
Make the slave memory node of state can adapter tube client data manipulation process, provide unremitting data manipulation clothes for client
Business, improves the availability of cluster file system.
Further, when primary storage node provides service for client, it is possible to break down, then primary storage node needs
Timing detects its link connection state and service running state of programs, to find failure in time, and makes respective handling, then base
The data processing method second embodiment of cluster file system of the present invention, in the present embodiment, reference are proposed in first embodiment
Fig. 8, the data processing method of the cluster file system further include:
Step S70, the memory node be main memory node when, the memory node detect its link connection state and
The operating status of the memory node；
Step S80 is run in the acquisition memory node real-time detection to its link connection failure or the memory node
When failure, the memory node chooses primary storage node in the slave memory node in normal operating conditions, by the institute of selection
Stating from the address mark of memory node is main memory node address；
The host node address of label is sent to described in control node and selection from memory node by step S90,
Wherein, the control node updates the host node address saved using the host node address received, and choose
When the memory node receives the host node address, working condition is switched to primary storage node state.
When needing to switch memory node, a slave memory node for being in normal operating conditions is chosen, is selected described
Slave service node address mark be new primary storage address of node, its address is sent to control node, control node
Client request later will be forwarded to new primary storage node, the slave memory node being selected can be switched to main memory
Node working condition is stored up, new primary storage node can read reading data or write-in progress information in its buffer area, directly
Corresponding process is taken over, the data manipulation process interrupt of client will not be made.
It is understood that may be roughly the same from the performance of memory node, it is also possible to which performance difference is larger, so in master
When memory node switches over, if all very nearly the same from memory node performance, one can be randomly selected in normal work
The slave memory node of state is as primary storage node, can be according to from storage if all performance differences from memory node are larger
The performance sequence of node selects a higher node of performance in normal operating conditions as new primary storage node.
The data processing method for the cluster file system that the present embodiment proposes, timing detect the link connection of primary storage node
State and service running state of programs, find the failure of primary storage node, and then switch primary storage node in time, mention for client
For unremitting reading and writing data service, it is ensured that the stability of cluster file system data processing service.
Referring to Fig. 9, the step of memory node detects the operating status of its link connection state and the memory node
Include:
Step S71, the memory node timing send the first detection data packet to the control node；
Step S72 receives the second response data packet that the control node is fed back based on the detection data packet:
Step S73 when receiving the response data packet within a preset time interval, judges the reading of the memory node
And whether write-in is normal；
Step S74 determines the memory node normal operation in the reading and normal write-in of the memory node；
Step S75 determines the memory node operation troubles in the reading and abnormal write-in of the memory node.
In the present embodiment, primary storage nodal test can not continue as clients providing data reading to its failure
Or when write service, primary storage node can select a slave memory node for being in normal operating conditions, will be selected from storage
Node is switched to primary storage node, the reading data or write-in process of adapter tube client.
Further, it is the safety for improving cluster file system, is also mentioned based on the first or second embodiments present invention
The 3rd embodiment of the data processing method of cluster file system out, referring to Fig.1 0, before the step S10, the cluster text
The data processing method of part system further comprises the steps of:
Step S100, when memory node receives file read or write request, the memory node determines the reading
It takes or whether the corresponding data of write request needs to verify；
In the present embodiment, when primary storage node receives file read or write request, the requested visit can be detected
The access level for the file asked, if the file is shared file, it can when being accessed by all users, then do not need to user
Identity authority verified, if the file is privately owned file, i.e., the described file can only be had the user of permissions
Access, then need to verify the identity authority of user, determine whether the file can be issued the client of access request
Access.
Step S110, when the corresponding data of the read or write request need to verify, the memory node, which obtains, to be used
The verification information of family input；
It is privately owned file in the file of client request access, that is, when needing to verify, primary storage node needs to obtain user
The information of input verifies the permission of user.At this point, user may carry out register, primary storage node only needs
The information of control node forwarding is traversed, data of the crawl with username and password keyword or handle, and then obtained
The verification information of user's input, verifies the permission of user；If user not yet logs at this time, primary storage node can be to control
Node sends message, and notice control node verifies prompting to client sending permission or control client pops up login interface,
After user's validation information, the verification information that the user inputs is forwarded to primary storage node by control node and is tested
Card.
Step S120, judges whether the verification information matches with the verification information prestored.
When the verification information is matched with the verification information prestored, the memory node is executed according to the read requests
And the solicited message in the buffer area reads the data in its memory block, or according to said write request and described slow
Deposit the step of data are written into its memory block for the solicited message in area, i.e. step S10.
The data processing method for the cluster file system that the present embodiment proposes can obtain after receiving data operation request
The operating right of the data is taken, if the data need Authority Verification, obtains the verification information of user's input, it is defeated in user
When the verification information entered is matched with the verification information prestored, corresponding data manipulation is just executed, improves cluster file system data
Safety.
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, method, article or the device that include a series of elements not only include those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or device institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including being somebody's turn to do
There is also other identical elements in the process, method of element, article or device.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in a storage medium
In (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that a terminal device (can be mobile phone, computer, cloud
Hold server, air conditioner or the network equipment etc.) execute method described in each embodiment of the present invention.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of data processing equipment of cluster file system, which is characterized in that the data processing of the cluster file system fills
It sets and includes:
Data processing module, for when memory node receives file read or write request, according to the read requests with
And the solicited message in buffer area reads the data in memory block, or according in said write request and the buffer area
Data are written into memory block for solicited message；
Memory module, for the solicited message in the read or write request to be stored in buffer area, the buffer area setting exists
On the memory node；
Synchronization module, for the solicited message to be synchronized to the slave memory node in cluster file system, the solicited message
I/O offset including process number and process.
2. the data processing equipment of cluster file system as described in claim 1, which is characterized in that the cluster file system
Data processing equipment further include:
Detection module when for receiving service request, detects the type of the service request, judge the service request whether be
Read or write request；
Respond module responds the service request if not being read or write request for the service request；
The data processing module, if being also used to the service request is read or write request, according to the read requests
And the solicited message in the buffer area reads the data in memory block, or according to said write request and the caching
Data are written into memory block for solicited message in area.
3. the data processing equipment of cluster file system as described in claim 1, which is characterized in that the cluster file system
Data processing equipment further include:
Determining module, for when receiving file read or write request, determining the corresponding number of the read or write request
According to whether needing to verify；
Module is obtained, for obtaining the verifying of user's input when the corresponding data of the read or write request need to verify
Information；
The data processing module is also used to when the verification information is matched with the verification information prestored, according to the reading
Request and the buffer area in solicited message read memory block in data, or according to said write request and it is described
Data are written into memory block for solicited message in buffer area.
4. the data processing equipment of cluster file system as claimed in claim 2, which is characterized in that the cluster file system
Data processing equipment further include:
The detection module is also used to detect link connection state and operating status；
Switching module, for when detecting link connection failure or operation troubles, in normal operating conditions from storage
In node choose primary storage node, by described in selection from the address mark of memory node be main memory node address；
Update module, for the primary storage node address of label to be sent to described in control node and selection from storage
Node, wherein the control node updates the primary storage node saved using the primary storage node address received
Address, and working condition is switched to primary storage section when receiving the primary storage node address by the memory node chosen
Dotted state.
5. the data processing equipment of cluster file system as claimed in claim 4, which is characterized in that the detection module packet
It includes:
Link detecting unit sends the first detection data packet to the control node for timing and receives the control node
The second response data packet based on detection data packet feedback；
Detection unit is read and write, when for receiving the response data packet within a preset time interval, judging to read and be written
It is whether normal；
Judging unit, when reading and being written abnormal, determines for determining normal operation when reading and being written normal
Operation troubles, and when not receiving the response data packet within a preset time interval, determine link connection failure.
6. a kind of data processing method of cluster file system, which is characterized in that the data processing side of the cluster file system
Method includes:
When memory node receives file read or write request, the memory node is according to the read requests and caching
Solicited message in area reads the data in its memory block, or according to the request in said write request and the buffer area
Data are written into its memory block in information；
Solicited message in the read or write request is stored in its buffer area by the memory node, and the buffer area setting exists
On the memory node；
The solicited message is synchronized to the slave memory node in cluster file system, the request packet by the memory node
Include the I/O offset of process number and process.
7. the data processing method of cluster file system as claimed in claim 6, which is characterized in that the memory node according to
Solicited message in the read requests and the buffer area reads the data in its memory block, or is asked according to said write
Ask and the buffer area in solicited message the step of data are written into its memory block before further include:
When the memory node receives service request, the type of the service request is detected；
Judge whether the service request is read or write request；
If the service request is read or write request, the memory node is executed according to read requests and described
Solicited message in buffer area reads the data in its memory block, or according in said write request and the buffer area
The step of data is written into its memory block for solicited message；
It is described judge the step of whether service request is read or write request after, the data of the cluster file system
Processing method further include:
If the service request is not read or write request, the service request is responded.
8. the data processing method of cluster file system as claimed in claim 6, which is characterized in that the memory node according to
Solicited message in the read requests and the buffer area reads the data in its memory block, or is asked according to said write
Ask and the buffer area in solicited message the step of data are written into its memory block before, the cluster file system
Data processing method further comprises the steps of:
When memory node receives file read or write request, the memory node determines the read or write request pair
Whether the data answered, which need, is verified；
When the corresponding data of the read or write request need to verify, the memory node obtains the verifying letter of user's input
Breath；
When the verification information is matched with the verification information prestored, execute the memory node according to the read requests and
Solicited message in the buffer area reads the data in its memory block, or according to said write request and the buffer area
In solicited message the step of data are written into its memory block.
9. the data processing method of cluster file system as claimed in claim 6, which is characterized in that the cluster file system
Data processing method further include:
When the memory node is main memory node, the memory node detects its link connection state and the memory node
Operating status；
It is described to deposit when obtaining the memory node real-time detection to its link connection failure or the memory node operation troubles
Store up node in normal operating conditions slave memory node in choose primary storage node, by described in selection from memory node
Address mark is main memory node address；
The primary storage node address of label is sent to described in control node and selection from memory node, wherein institute
It states control node and updates the primary storage node address saved using the primary storage node address received, and choose
When the memory node receives the primary storage node address, working condition is switched to primary storage node state.
10. the data processing method of cluster file system as claimed in claim 9, which is characterized in that the memory node inspection
The step of surveying the operating status of its link connection state and the memory node include:
The memory node timing sends the first detection data packet to the control node；
Receive the second response data packet that the control node is fed back based on the detection data packet:
When receiving the response data packet within a preset time interval, judge the memory node reading and write-in whether
Normally；
In the reading and normal write-in of the memory node, the memory node normal operation is determined, save in the storage
When the reading and abnormal write-in of point, the memory node operation troubles is determined；
When not receiving the response data packet within a preset time interval, the link connection failure of the memory node is determined.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510847053.1A CN105511805B (en) | 2015-11-26 | 2015-11-26 | The data processing method and device of cluster file system |
| PCT/CN2016/105219 WO2017088664A1 (en) | 2015-11-26 | 2016-11-09 | Data processing method and apparatus for cluster file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510847053.1A CN105511805B (en) | 2015-11-26 | 2015-11-26 | The data processing method and device of cluster file system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105511805A CN105511805A (en) | 2016-04-20 |
| CN105511805B true CN105511805B (en) | 2019-03-19 |
Family
ID=55719825
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510847053.1A Active CN105511805B (en) | 2015-11-26 | 2015-11-26 | The data processing method and device of cluster file system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN105511805B (en) |
| WO (1) | WO2017088664A1 (en) |
Families Citing this family (31)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105511805B (en) * | 2015-11-26 | 2019-03-19 | 深圳市中博科创信息技术有限公司 | The data processing method and device of cluster file system |
| CN106254103B (en) * | 2016-07-28 | 2019-08-16 | 北京国电通网络技术有限公司 | A kind of dynamically configurable method and device of RTMP group system |
| CN106598762B (en) * | 2016-12-29 | 2020-04-17 | 上海理想信息产业（集团）有限公司 | Message synchronization method |
| CN108512753B (en) | 2017-02-28 | 2020-09-29 | 华为技术有限公司 | Method and device for transmitting messages in cluster file system |
| CN107168649B (en) * | 2017-05-05 | 2019-12-17 | 南京城市职业学院 | method and device for data distribution in distributed storage system |
| CN107707620B (en) * | 2017-08-30 | 2020-09-11 | 华为技术有限公司 | Method and device for processing IO (input/output) request |
| CN107704201B (en) * | 2017-09-11 | 2020-07-31 | 厦门集微科技有限公司 | Data storage processing method and device |
| CN109543204B (en) * | 2017-09-22 | 2022-09-13 | 南京理工大学 | Electric heating integrated analysis method for semiconductor device under human body electrostatic action |
| CN108023772B (en) * | 2017-12-07 | 2021-02-26 | 海能达通信股份有限公司 | Abnormal node repairing method, device and related equipment |
| CN110099084B (en) * | 2018-01-31 | 2021-06-15 | 北京易真学思教育科技有限公司 | Method, system and computer readable medium for ensuring storage service availability |
| CN108829720B (en) * | 2018-05-07 | 2022-01-14 | 麒麟合盛网络技术股份有限公司 | Data processing method and device |
| CN110716692B (en) * | 2018-07-13 | 2022-11-25 | 浙江宇视科技有限公司 | Reading performance improving method and device, storage node and data reading method |
| CN109213507A (en) * | 2018-08-27 | 2019-01-15 | 郑州云海信息技术有限公司 | A kind of upgrade method and server |
| CN109508317B (en) * | 2018-10-31 | 2023-06-09 | 陕西合友网络科技有限公司 | High-capacity data and service management system |
| CN109587221A (en) * | 2018-11-09 | 2019-04-05 | 平安科技（深圳）有限公司 | Large data sets group managing means, device, storage medium and computer equipment |
| CN110474981A (en) * | 2019-08-13 | 2019-11-19 | 中科天御(苏州)科技有限公司 | A kind of software definition dynamic security storage method and device |
| CN110674095B (en) * | 2019-09-27 | 2022-06-10 | 浪潮电子信息产业股份有限公司 | CTDB cluster expansion method, device and equipment and readable storage medium |
| CN110868323B (en) * | 2019-11-15 | 2022-07-22 | 浪潮电子信息产业股份有限公司 | Bandwidth control method, device, equipment and medium |
| CN111314129B (en) * | 2020-02-13 | 2023-05-12 | 上海凯岸信息科技有限公司 | High-availability architecture based on file type storage service |
| CN111371865B (en) * | 2020-02-26 | 2023-02-24 | 上海达梦数据库有限公司 | Client connection relation adjusting method, system and node |
| CN112468330B (en) * | 2020-11-13 | 2022-12-06 | 苏州浪潮智能科技有限公司 | Method, system, equipment and medium for setting fault node |
| CN112395165A (en) * | 2020-11-27 | 2021-02-23 | 中电科技（北京）有限公司 | Method and equipment for acquiring resource utilization rate of server and server |
| CN112671905B (en) * | 2020-12-23 | 2023-04-28 | 广州三七互娱科技有限公司 | Service scheduling method, device and system |
| CN112737962B (en) * | 2020-12-24 | 2023-06-02 | 平安科技（深圳）有限公司 | Storage service request processing method and device, computer equipment and storage medium |
| CN113051349A (en) * | 2021-04-02 | 2021-06-29 | 广东美电贝尔科技集团股份有限公司 | Data synchronization method for duty system |
| CN113259092A (en) * | 2021-04-04 | 2021-08-13 | 余绍祥 | Document distributed encryption system |
| CN112988905B (en) * | 2021-04-27 | 2021-08-10 | 北京沃丰时代数据科技有限公司 | Node memory synchronization method and device for cluster deployment |
| CN113590040B (en) * | 2021-07-29 | 2024-03-19 | 郑州阿帕斯数云信息科技有限公司 | Data processing method, device, equipment and storage medium |
| CN116266152A (en) * | 2021-12-16 | 2023-06-20 | 中移(苏州)软件技术有限公司 | Information processing method and device and storage medium |
| CN114189547B (en) * | 2022-02-14 | 2022-05-03 | 北京安盟信息技术股份有限公司 | SSL tunnel fast switching method under cluster |
| CN116644039B (en) * | 2023-05-25 | 2023-12-19 | 安徽继远软件有限公司 | Automatic acquisition and analysis method for online capacity operation log based on big data |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4643543B2 (en) * | 2006-11-10 | 2011-03-02 | 株式会社東芝 | Storage cluster system with cache consistency guarantee function |
| CN103207841A (en) * | 2013-03-06 | 2013-07-17 | 青岛海信传媒网络技术有限公司 | Method and device for data reading and writing on basis of key-value buffer |
| CN103207894A (en) * | 2013-03-14 | 2013-07-17 | 深圳市知正科技有限公司 | Multipath real-time video data storage system and cache control method thereof |
| CN103634269A (en) * | 2012-08-21 | 2014-03-12 | 中国银联股份有限公司 | A single sign-on system and a method |
| US8875146B2 (en) * | 2011-08-01 | 2014-10-28 | Honeywell International Inc. | Systems and methods for bounding processing times on multiple processing units |
| CN104361030A (en) * | 2014-10-24 | 2015-02-18 | 西安未来国际信息股份有限公司 | Distributed cache architecture with task distribution function and cache method |
| CN104580432A (en) * | 2014-12-23 | 2015-04-29 | 上海帝联信息科技股份有限公司 | Memcached system, memory cache data provision method and device, memory cache data maintenance method and device as well as cluster maintenance method and device |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110055494A1 (en) * | 2009-08-25 | 2011-03-03 | Yahoo! Inc. | Method for distributed direct object access storage |
| CN102035862B (en) * | 2009-09-30 | 2013-11-06 | 国际商业机器公司 | Configuration node fault transfer method and system in SVC cluster |
| US8751641B2 (en) * | 2011-09-15 | 2014-06-10 | International Business Machines Corporation | Optimizing clustered network attached storage (NAS) usage |
| US9189510B2 (en) * | 2013-02-26 | 2015-11-17 | Facebook, Inc. | System and method for implementing cache consistent regional clusters |
| CN105511805B (en) * | 2015-11-26 | 2019-03-19 | 深圳市中博科创信息技术有限公司 | The data processing method and device of cluster file system |
-
2015
- 2015-11-26 CN CN201510847053.1A patent/CN105511805B/en active Active
-
2016
- 2016-11-09 WO PCT/CN2016/105219 patent/WO2017088664A1/en active Application Filing
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4643543B2 (en) * | 2006-11-10 | 2011-03-02 | 株式会社東芝 | Storage cluster system with cache consistency guarantee function |
| US8875146B2 (en) * | 2011-08-01 | 2014-10-28 | Honeywell International Inc. | Systems and methods for bounding processing times on multiple processing units |
| CN103634269A (en) * | 2012-08-21 | 2014-03-12 | 中国银联股份有限公司 | A single sign-on system and a method |
| CN103207841A (en) * | 2013-03-06 | 2013-07-17 | 青岛海信传媒网络技术有限公司 | Method and device for data reading and writing on basis of key-value buffer |
| CN103207894A (en) * | 2013-03-14 | 2013-07-17 | 深圳市知正科技有限公司 | Multipath real-time video data storage system and cache control method thereof |
| CN104361030A (en) * | 2014-10-24 | 2015-02-18 | 西安未来国际信息股份有限公司 | Distributed cache architecture with task distribution function and cache method |
| CN104580432A (en) * | 2014-12-23 | 2015-04-29 | 上海帝联信息科技股份有限公司 | Memcached system, memory cache data provision method and device, memory cache data maintenance method and device as well as cluster maintenance method and device |
Non-Patent Citations (3)
| Title |
|---|
| ISCSI双控制器存储系统缓存设计与实现;吴昌松;《万方数据》;20141028;1-67 |
| RAID系统的多控制器技术研究;王光辉;《万方数据》;20040408;正文第19-21、39-42页第1-2节、第44-47页第3节，图4-1 |
| RAID系统的多控制器技术研究;王光辉;《万方数据》;20040408;正文第19-21、39-42页第1-2节、第48-49页第4节，图4-1 |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2017088664A1 (en) | 2017-06-01 |
| CN105511805A (en) | 2016-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105511805B (en) | The data processing method and device of cluster file system | |
| CN116302719B (en) | System and method for enabling high availability managed failover services | |
| US8271632B2 (en) | Remote access providing computer system and method for managing same | |
| CN101102288B (en) | A method and system for realizing large-scale instant message | |
| US7640451B2 (en) | Failover processing in a storage system | |
| US8572431B2 (en) | Disaster recovery framework | |
| CN110224871A (en) | A kind of high availability method and device of Redis cluster | |
| CN108234191A (en) | The management method and device of cloud computing platform | |
| CN105721200A (en) | Master-slave server system application method and system thereof | |
| US9152441B2 (en) | Systems and methods involving virtual machine host isolation over a network via a federated downstream cluster | |
| JP2007528085A (en) | System and method for providing a channel in an application server and transaction based system | |
| JP2013187707A (en) | Host provision system and communication control method | |
| CN108959045A (en) | A kind of test method and system of NAS clustering fault performance of handoffs | |
| CN110908723A (en) | Main/standby switching method and device of operating system and related equipment | |
| CN110855488A (en) | Virtual machine access method and device | |
| CN109787865B (en) | Method, system, switch and storage medium for verifying upgrading condition | |
| CN110061876B (en) | Optimization method and system of operation and maintenance auditing system | |
| CN108366087B (en) | ISCSI service realization method and device based on distributed file system | |
| CN106358140B (en) | Contact person grouping method and device | |
| US20080198740A1 (en) | Service take-over system of multi-host system and method therefor | |
| JP2015114952A (en) | Network system, monitoring control unit, and software verification method | |
| Cisco | Configuration Fundamentals Configuration Guide Cisco IOS Release 12.0 | |
| CN114915545B (en) | Application scheduling deployment management method based on DHCP network cluster | |
| CN110266795A (en) | One kind being based on Openstack platform courses method | |
| EP3018931B1 (en) | Method and system for establishing a self-organized mobile core in a cellular communication network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200918 Address after: 100089 Floor 1-4, No. 2 Building, No. 9 Courtyard, Dijin Road, Haidian District, Beijing Patentee after: BEIJING TOYOU FEIJI ELECTRONICS Co.,Ltd. Address before: 518000 Guangdong city of Shenzhen province Nanshan District South Road Fiyta Technology Building Room 1402 Patentee before: SHENZHEN ZHONGBO KECHUANG INFORMATION Co.,Ltd. |
|
| TR01 | Transfer of patent right |