CN109347906A - A kind of data transmission method, device and server - Google Patents
A kind of data transmission method, device and server Download PDFInfo
- Publication number
- CN109347906A CN109347906A CN201811008654.3A CN201811008654A CN109347906A CN 109347906 A CN109347906 A CN 109347906A CN 201811008654 A CN201811008654 A CN 201811008654A CN 109347906 A CN109347906 A CN 109347906A
- Authority
- CN
- China
- Prior art keywords
- request
- node
- log
- user
- host node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0668—Management of faults, events, alarms or notifications using network fault recovery by dynamic selection of recovery network elements, e.g. replacement by the most appropriate element after failure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/30—Decision processes by autonomous network management units using voting and bidding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/10—Active monitoring, e.g. heartbeat, ping or trace-route
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Health & Medical Sciences (AREA)
- Cardiology (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention proposes a kind of data processing method, device and server, and described method includes following steps: receiving user's request that client is sent；It is requested with user described in logged, and reads the log after the recording is completed；The network transmitting-receiving of progress strong consistency in the case where the asynchronous remote of abstract network layer calls service framework；State machine engine is called to receive strong consistency readjustment, so that the state machine engine executes user's request.
Description
Technical field
The present invention relates to database technical field, in particular to a kind of data processing method, device and server.
Background technique
The leader follower replication of Redis uses asynchronous replication method, periodically reports duplication stream to primary server from server
Processing progress.
Redis is a key-value storage system.Similar with Memcached, it supports the value type phase of storage
To more, including string (character string), list (chained list), set (set), zset (sorted set-- ordered set) and
Hash (hash type).These data types all support push/pop, add/remove and take intersection union and difference set and Geng Feng
Rich operation, and these operations are all atomicities.On this basis, Redis supports the sequence of various different modes.With
Memcached is the same, and for guaranteed efficiency, data are all to cache in memory.Difference is that people Redis can be periodically more
Modification is operated the additional record file of write-in by new data write-in disk, and realizes Master- on this basis
Slave (principal and subordinate) is synchronous.
Redis supports master-slave synchronisation.Data can be synchronized from server from primary server to any number of, from service
Device, which can be, is associated with other primary servers from server.This makes Redis that single layer tree copying can be performed.Deposit can intentional nothing
Meaning carries out write operation to data.Due to fully achieving publish/subscribe mechanism, so that from database anywhere synchronization tree
When, a channel can be subscribed to and receive the complete news release record of primary server.Synchronize to the scalability of read operation and
Data redundancy is helpful.
However, Redis does not provide strongly consistent synchronous mode, can not cover in the financial levels application scenarios such as bank, insurance.
Summary of the invention
In order to solve the technical problems existing in the prior art, the embodiment of the invention provides a kind of data processing method,
Device, server and storage medium.The technical solution is as follows:
On the one hand, a kind of data processing method is provided, comprising: receive user's request that client is sent；In the form of log
User's request is recorded, and reads the log after the recording is completed；Service frame is called in the asynchronous remote of abstract network layer
The network transmitting-receiving of strong consistency is carried out under frame；State machine engine is called to receive strong consistency readjustment, so that the state power traction
It holds up and executes user's request.
On the one hand, a kind of data processing equipment is provided, comprising: receiving module, the user for receiving client transmission ask
It asks；Log read module for requesting with user described in logged, and reads the log after the recording is completed；By force
Consistent transceiver module, for carrying out the network transmitting-receiving of strong consistency in the case where the asynchronous remote of abstract network layer calls service framework；
Calling module, for calling state machine engine to receive strong consistency readjustment, so that the state machine engine executes the user
Request.
On the other hand, a kind of server is provided, the server includes device above-mentioned.
On the other hand, a kind of storage medium is provided, at least one instruction, at least one section of journey are stored in the storage medium
Sequence, code set or instruction set, at least one instruction, at least one section of program, the code set or the instruction set are by handling
Device is loaded and is executed to realize data processing method above-mentioned.
Technical solution bring beneficial effect provided in an embodiment of the present invention includes: that can be guaranteed by strong consistency transmitting-receiving
The high security and high reliability of data processing guarantee that data handling procedure can be used for the financial levels applied field such as bank, insurance
Scape and other scenes for needing strict safety grade.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment
Attached drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the invention, for
For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other
Attached drawing.
Fig. 1 is a kind of distributed data management cell schematics provided in an embodiment of the present invention；
Fig. 2 is the network architecture schematic diagram provided in an embodiment of the present invention using public strong consistency algorithm and log library；
Fig. 3 is the network architecture schematic diagram provided in an embodiment of the present invention with using public strong consistency algorithm and log library
Corresponding distributed data management cell schematics；
Fig. 4 is data processing method steps flow diagram provided in an embodiment of the present invention；
Fig. 5 is the interactive process schematic diagram of data processing processing method provided in an embodiment of the present invention；
Fig. 6 is heartbeat packet transmitting path schematic diagram provided in an embodiment of the present invention；
Fig. 7 is that heartbeat packet provided in an embodiment of the present invention sends interactive process schematic diagram；
Fig. 8 is node life cycle schematic diagram provided in an embodiment of the present invention；
Fig. 9 is data processing equipment functional block diagram provided in an embodiment of the present invention；
Figure 10 is receiving module functional block diagram provided in an embodiment of the present invention；
Figure 11 is log read module principle block diagram provided in an embodiment of the present invention；
Figure 12 is strongly consistent transceiver module functional block diagram provided in an embodiment of the present invention；
Figure 13 is master-slave synchronisation module principle block diagram provided in an embodiment of the present invention；
Figure 14 is the functional block diagram of calling module provided in an embodiment of the present invention；
Figure 15 is the data processing equipment functional block diagram provided in an embodiment of the present invention comprising result return module；
Figure 16 is other optional submodule functional block diagrams of strongly consistent transceiver module provided in an embodiment of the present invention；
Figure 17 is server architecture schematic diagram provided in an embodiment of the present invention.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with attached drawing to embodiment party of the present invention
Formula is described in further detail.
In one embodiment of the invention, as shown in Figure 1, proposing a kind of distributed data management unit, each data
Administrative unit is known as fragment (shard), and Fig. 1 shows the fragment situation on a machine.Each traffic ID (Bid) includes one
A copy set (replicat set), includes several fragments (shard) in a copy set again, and each management by district is centainly big
Small shared drive block, bottom engine fastly organize these memories the function of having realized each interface of Redis.By fragment
ID (shard id) can calculate the CPU ID where service belonging to fragment.The fragment of each copy set is distributed in not in principle
Same machine, even entire copy is concentrated with 3 machines A, B, C at this time, when creating a Bid, respectively creates one in A, B, C respectively
A fragment, and after specifying active and standby role, user can be received and request and service.
According to above description, different the first fragments of fragment (shard-0), the second fragment (shard-1), in Fig. 1
Three fragments (shard-2), the 4th fragment (shard-3) belong to different business, and user's request is to be sent at random arbitrarily
CPU's, find oneself not to be target fragment by query routing if at this moment receiving service belonging to the CPU of user's request,
Request is sent to target CPU by submitting function (submit), is handled by the service on target CPU.Each fragment is also
Corresponding respective the first shared drive of shared drive (shm-0), the second shared drive (shm-1), third shared drive (shm-
2), the 4th shared drive (shm-3), shared drive is for storing key content (key value).It is the request generation of forwarding below
Chip segment:
Return submit(ctx->CPU(),[this,ctx]){
Process(ctx)；
})
In some financial business scenes, the coherence request of copy set each for data is very high, and framework shown in FIG. 1 is also
The strongly consistent characteristic between copy can not be kept completely.
In one embodiment, it as shown in Fig. 2, in order to guarantee compatibility, is not directly based upon asynchronous remote and calls service
Frame (RPC) realizes daily record reproduction and consistency algorithm, but uses the public strong consistency algorithm of interior structure and log library
(libraft) realize that basic logic in daily record reproduction and consistency algorithm agreement and interface are called.Public strong consistency is calculated
Method and log library (libraft) provide conventional sync cap, since asynchronous remote calls service framework that can not call directly resistance
The disk I/O and network I/O of plug, so the realization difficult point of strongly consistent logic essentially consists in the knot of asynchronous frame and sync cap
It closes.Therefore, in the logic of strongly consistent, induction disk IO, network I/O and the shape of service framework (RPC) are called based on asynchronous remote
State power traction is held up, and these asynchronous implements are packaged into sync cap and are returned for public strong consistency algorithm and log library (libraft)
It adjusts, the strongly consistent duplication of principal and subordinate's fragment may be implemented in this way.
With corresponding to Fig. 2, Fig. 3 illustrates a kind of distributed data management unit, in conjunction with Fig. 2 and Fig. 3, in the identical of fragment
Rank increases a strong consistency fragment object.Strong consistency fragment object can be considered as in a strong consistency transport protocol
Node (Node), the complete function with individual node in strong consistency transport protocol, in each strong consistency fragment packets
Comprising a public strong consistency algorithm and log library (libraft), sync cap, disk comprising strong consistency basic logic
It realizes with network I/O logic, state machine engine (FSM), connect for what public strong consistency algorithm and log library (libraft) were adjusted back
Mouthful.Node (Node) in each strong consistency transport protocol has the role of oneself, in host node, both candidate nodes, from section
One in point.Host node receives user's read-write requests, only receives read request from node, understands from after receiving write request request
Host node is sent in such a way that asynchronous remote calls service framework (RPC).After host node receives read request, first request sequence
Column are melted into a log and carry out rule operation, the log of rule can then be read from disk, by network I/O this
Log is synchronized to other from node, other will do it rule after receiving the synchronous request of log from node, when this log
Through success rule, host node is notified that from node.Finally, if host node finds that this log is successfully written by most nodes
Journal file will call bottom engine, while being also notified that from node and calling bottom engine, using corresponding to this log
Then write operation returns result to user.
In above process, user's request arrives first at the node (Node) in strong consistency transport protocol, in strongly consistent
Property node between circulate, confirmation log have arrived at most number of nodes after, just calling bottom engine.
Request path of user's request in single strong consistency fragment is as shown in Figure 2:
Client (Client), after receiving request from network, package is sent to strong consistency module；
Timer (Timer) drives public strong consistency module, periodically with response timeout event；
Abstract network layer (Network) provides the network transmission-receiving function that asynchronous remote calls service framework (RPC), and drives
Dynamic strong consistency drive module；
State machine engine (FSM) receives the readjustment from strong consistency, and user's request is applied in actual engine, and
Return to user's result；
Public strong consistency algorithm and log library (libraft), it is synchronous with log library agreement to realize consistency algorithm
Interface is driven by remote call service frame；
Disk read-write logic (DiskLog) handles the log read-write requests from public strong consistency algorithm and log library.
User requests with recovery process abstract network layer (Network), state machine engine as shown in figure 4, in Fig. 4
(FSM), disk read-write logic (DiskLog) is all to call the asynchronous interface of service framework (RPC) to be packaged into together asynchronous remote
Step interface is supplied to public strong consistency algorithm and log library (libraft) is called, when asynchronous remote calls service framework (RPC)
When needing to call the sync cap of public strong consistency algorithm and log library (libraft), such as when there is user's request to come in, just
It needs to call the spy called in service framework (RPC) in public strong consistency algorithm and log library (libraft) in asynchronous remote
Stationary interface is requested to receive user, such as user data receiving interface (RecvUserData).Specifically, Fig. 4 describes as follows
Data processing step:
Step S401 receives user's request that client is sent.
Client receives user's request, and public strong consistency algoritic module will be sent to after the Request Packet.It is public strong
Consistency algorithm module includes the node of Three role: master node, slave node and both candidate nodes, is received by host node and comes from client
The package at end.
Step S402 is requested with user described in logged, and reads user request after the recording is completed；
After host node in public strong consistency algoritic module receives read request, request sequence is first melted into a log,
Then the request of log rule is sent to disk read-write logic, after rule completion, host node reads the log of the rule.
Step S403, the network of progress strong consistency in the case where the asynchronous remote of abstract network layer calls service framework (RPC)
Transmitting-receiving.
The log of reading is synchronized to other from node by network I/O by public strong consistency algoritic module, other are from node
After receiving log synchronization request, rule operation can be carried out to request and be notified that if day aims at successfully rule from node
Host node.
Step S404 calls state machine engine (FSM) to receive strong consistency readjustment, so that the state machine engine executes
User's request.
When host node discovery log be successfully written journal file by most of nodes, the state machine of bottom will be called
Engine, while being also notified that from node and bottom engine is called then to return to result using write operation corresponding to this log
To user.
In conclusion strong consistency can bring high security and high reliability, guarantee that data handling procedure can be used for
The financial levels such as bank, insurance application scenarios and other scenes for needing strict safety grade.Fig. 5 describes above-mentioned steps
Interaction flow.
It is called in service framework (RPC) in asynchronous remote, can uniquely call the place of sync cap is in RPC frame line
Otherwise Cheng Zhong will lead to abnormal program termination (coredump).So starting one is asynchronous in the scene for receiving user's request
Remote call service frame (RPC) thread, calls the special interface in public strong consistency algorithm and log library (libraft),
Such as RecvUserData ().Each asynchronous remote call service frame (RPC) thread is run on the CPU for creating it, and not
Shared stack, so not needing lock to guarantee thread-safe.The similar above asynchronous remote calls service framework (RPC) thread to exist
There are several during the realization of strongly consistent, either asynchronous remote calls service framework (RPC) that public strong consistency is called to calculate
The sync cap or public strong consistency algorithm of method and log library (libraft) and the readjustment of log library (libraft) are all
What is wherein carried out.For example, can be by future interface encapsulation at sync cap, to meet the synchronous encapsulation of asynchronous interface:
Future<>DisklogImp_::DisklogImpl_::AppendEntry(const Entry&entry)
Return gate(_gate,[this,entry]){
Return lock(_lock,for_write(),[this,entry]){
Void DisklogImp_::DisklogImpl_::AppendEntry(const Entry&entry){
_impl.AppendEntry(entry).get()；
}
It is that an asynchronous method is packaged into sync cap above, get () function here waits asynchronous
The future that AppendEntry () is returned becomes completion status.After the completion of future state becomes, other on this CPU
RPC frame thread can continue to execute.As it can be seen that the interface of asynchronous execution frame can be synchronized by above-mentioned steps
Encapsulation.
In strong consistency public library object (libraft), server is endowed Three role: master node, slave node and time
Select node.In these three nodes, the status of host node is core, is used to receive the request from client, client
It will not transmit the request to from node, even if client is transmitted the request to from node, the request of client can be refused from node,
And the IP address of host node is returned into client.Host node and be one-way flow from the data between node, data are with the heart
It includes journal file from node, heartbeat packet that the form for jumping packet is flowed to by host node.Carry out the heart of autonomous node from node reception
Packet is jumped, if one does not receive the heartbeat packet of host node from node before the deadline, which can be changed into candidate section
Point, and initiate ballot request to other from node, when it is most of vote from node to the both candidate nodes when, both candidate nodes transformation
For host node.As both candidate nodes, the heartbeat packet of host node transmission is not received.
Under the driving of timer (Timer), host node can be synchronous when not having to need to respectively sending heartbeat packet from node
When log, the content of heartbeat packet is sky, log that is on the contrary then taking needs synchronization.Host node judges the timeout case from node
It is operated to do some disaster tolerances, then can judge whether host node is overtime according to the interval for receiving heartbeat packet from node, if host node is super
When then oneself become both candidate nodes initiate election.Heartbeat packet transmitting path is as shown in Figure 6:
Step S601, host node send heartbeat packet to from node；When there is no to need synchronous log, the content of heartbeat packet
For sky；When needing synchronization log, then takes and need synchronous log.
Step S602, host node judge the timeout case from node to carry out disaster tolerance operation.
Step S603 judges whether host node is overtime from node according to the interval for receiving heartbeat packet；When host node, from section
Point then becomes both candidate nodes and initiates election application.
Fig. 7 then illustrates the interactive process of heartbeat packet transmission.
In a specific embodiment, runing time module (run_ is called from the timer period of node
Periodic), which can check the heart time that last time host node sends over, if not receiving the heartbeat of host node for a long time
Oneself, will be become both candidate nodes from node and initiate to elect by (being more than time threshold).Both candidate nodes pass through the presently written phase
Module and current ballot factor module are limited to adjust back the election metamessage of the persistence in local disk logic.Both candidate nodes can lead to
It crosses and reads last directory information module and read time limit module readjustment disk logic to obtain itself time limit and index.Time limit and
Index is for electing request, other nodes decide whether to agree to ballot according to the time limit and index of election node in election process.
Both candidate nodes initiate election by sending other nodes of instruction readjustment abstract network layer into cluster, and both candidate nodes pass through reception
Function adjusts back time packet that abstract network receives election request.As shown in figure 8, both candidate nodes are having time limitation, this realization
Limitation is reflected as the time limit of both candidate nodes itself.The life cycle of both candidate nodes includes election phase, division ballot phase and general behaviour
Make the phase.In the election phase, both candidate nodes invite ticket to request to from node initiation；During division ballot, both candidate nodes, which receive, to be come since section
The ballot of point；The general operation phase is then to judge whether the both candidate nodes become host node according to index and node time limit, in addition, such as
There are two both candidate nodes to initiate ballot simultaneously for fruit, realizes then which both candidate nodes is competed as host node by the two.Two times
Select node one section of timeout such as 300ms waiting after because the poll that both sides obtain is the same, then 300ms with
Afterwards, then by the sending of the two candidates ticket is invited, at this moment the probability of gained vote substantially reduces simultaneously, then issuing the candidate for inviting ticket first
Person has obtained most of agreements, becomes host node, and another both candidate nodes was issued later when inviting ticket, those from node
It votes to first both candidate nodes, cannot vote again to it, it just becomes losing candidate, this last losing candidate also becomes common
From node a member.
In a possible embodiment, for user data receiving interface, consider following scene, server is to visitor
Family end sends data " _ META_DATA_ r n_USER_DATA ", it is desirable that the data _ META_DATA_ of " r n " before is first
It is received in secondary request, the data of remaining request call reading _ part USER_DATA_.Because tcp agreement is stream protocol
(Stream), and _ META_DATA_ data are not fixed length, thus have no idea guarantee a request call do not read _
The data of the part USER_DATA_, except non-once reads a character.At this point it is possible to consider the data peak parameter in request function
(MSG_PEEK), the prototype of request is ssize_t Recv (int s, void*buf, size_t len, int flags)；Usually
Flags is both configured to 0, and request function is read in the data to data caching in tcp caching at this time, and is removed from tcp caching
The data read.Flags is set as data peak parameter (MSG_PEEK), only tcp cache in reading data to caching
In, the data read are not removed from tcp caching, calls Recv still can read the data read just now again.
For scenario above, Recv (fd, buf, nbuf, MSG_PEEK) checks data, checks the position pos of " r n ", then Recv
(fd, buf, pos+2,0) reads (and removal) data.Certainly, such situation is more extreme, many times, even if primary
_ the data of the part USER_DATA_ have been read in Recv, and still first this partial data can be saved, be added to subsequent
Before Recv data, but if different Recv crosses over many functions, saves data and bring additional complexity.
In a possible embodiment, it is also contemplated that text protocol and two is supported in scene in addition, the same port
System agreement has a look several leading a character using data peak parameter (MSG_PEEK), and first judgement is text protocol or binary system
Agreement, then do request distribution and good selection.Certainly, it is called before real Recv and uses data peak parameter (MSG_
PEEK additional linear function) is caused to call.
In a possible embodiment, it is called in service framework (RPC) in asynchronous remote, uniquely can call to synchronize and connect
The place of mouth is otherwise to will lead to abnormal program termination (coredump) in the thread of frame.Each asynchronous remote call service
Frame (RPC) thread is run on the CPU for creating it, and does not share stack, so not needing lock to guarantee thread-safe.By
It is called in asynchronous remote and does not share stack between each thread in service framework (RPC), constituted between each thread similar
(channel) relationship in channel.Under general modfel, if the serial cyclical function (loop) twice that executes can generate continuous two
Secondary output, such as:
func loop(){
For i:=0；i<10；i++{
Fmt.Printf (" %d ", i)
}}
func main(){
loop()
loop()
}
So implementing result is 01234567890123456789.
And called in service framework (RPC) in asynchronous remote, cyclical function (loop) twice is executed if serial, only can
Export primary result 0123456789.This is because second of circulation does not have enough time also executing, and principal function is just under asynchronous frame
It has logged out.Therefore, under asynchronous frame, principal function is premature in order to prevent is exited, using the method for waiting, i.e., in loop
Increase the waiting time in () function:
func main(){
loop()；
loop()；
time.sleep(time.second)；
}
But the mode for increasing the waiting time undoubtedly will increase entire execution time consumption, it is possible to increase in cross-thread
Add notification function, and be that another thread increases obstruction:
for thread in threads；
thread.join()；
Only after channel, which executes, to be completed, it is just notified that elimination obstruction, and this mode has the possibility for bringing deadlock.Cause
This, calls service framework (RPC) as the place that can uniquely call sync cap asynchronous remote, it is possible to reduce due to other
Abortive possibility caused by calling.
In one embodiment of the invention, described as shown in figure 9, providing a kind of functional block diagram of data processing equipment
Device includes: receiving module, for receiving user's request of client transmission；Log read module, for being remembered in the form of log
User's request is recorded, and reads the log after the recording is completed；Strongly consistent transceiver module, in the different of abstract network layer
Walk the network transmitting-receiving that strong consistency is carried out under remote call service frame (RPC)；Calling module, for calling state machine engine
(FSM) strong consistency readjustment is received, so that the state machine engine executes user's request.
In an alternative embodiment, as shown in Figure 10, the functional block diagram of receiving module is provided, comprising: receive submodule
Block when receiving the request of the user from client for host node, receives user's request from client；Reject submodule
Block when for receiving the request from client from node, refuses the request, and return to the host node to client
Network address.
In an alternative embodiment, as shown in figure 11, the functional block diagram of log read module is provided, comprising: user
Request receiving submodule, for receiving user's request；Submodule is serialized, for user's request sequence to be turned to request day
Will；Rule requests submodule, for sending the request of log rule to disk read-write logic；Submodule is read, reception of magnetic disc is used for
The information that logic returns is read and write, when rule is completed, reads the rule log.
In an alternative embodiment, as shown in figure 12, the functional block diagram of strongly consistent transceiver module is provided, comprising: main
From synchronous submodule, for the log of reading being synchronized to host node from node；Rule notifies submodule, for described from section
Point carries out rule operation to the log, and in success rule, notifies the host node.
In an alternative embodiment, as shown in figure 13, the functional block diagram of master-slave synchronisation submodule is provided, comprising: the heart
Packet sending submodule is jumped, sends heartbeat packet to from node for host node；When there is no to need synchronous log, heartbeat packet it is interior
Hold for sky；When needing synchronization log, then takes and need synchronous log；Host node time-out judging submodule is used for host node
The timeout case from node is judged to carry out disaster tolerance operation；From node time-out judging submodule, from node according to receiving heartbeat packet
Interval judge whether host node overtime；When host node time-out, then become both candidate nodes from node, and initiates election application.
In an alternative embodiment, as shown in figure 14, the functional block diagram of calling module is provided, comprising: call submodule
Block, for most of when being successfully written journal file and notifying, calling state machine engine from node when host node receives
(FSM), and the state machine engine (FSM) is notified to carry out write operation to the log；As a result submodule is returned, is used for result
Return to client.
In an alternative embodiment, as shown in figure 15, it is former to provide the data processing equipment comprising result return module
Manage block diagram: result return module is used for after the state machine engine executes user's request, and implementing result is returned to institute
State client.
In an alternative embodiment, as shown in figure 16, other optional submodules of strongly consistent transceiver module are provided, are wrapped
Include: thread waits module, for carrying out the net of strong consistency in the case where the asynchronous remote of abstract network layer calls service framework (RPC)
During network transmitting-receiving, the parameter state of single thread return is waited；Parameter state monitoring module is used for monitoring parameter state；Together
Package module is walked, for after the completion of parameter state becomes, continuing to execute other lines that asynchronous remote calls service framework (RPC)
Journey.It can be to synchronize by the interface encapsulation in asynchronous frame by above procedure
Figure 17 is please referred to, it illustrates the structural schematic diagrams of server provided by one embodiment of the present invention.The server
Data processing method for the server side for implementing to provide in above-described embodiment.Specifically:
The server 1200 includes 1202 He of central processing unit (CPU) 1201 including random access memory (RAM)
The system storage 1204 of read-only memory (ROM) 1203, and connection system storage 1204 and central processing unit 1201
System bus 1205.The server 1200 further includes that the substantially defeated of information is transmitted between each device helped in computer
Enter/output system (I/O system) 1206, and is used for storage program area 1213, application program 1214 and other program modules
1215 mass-memory unit 1207.
The basic input/output 1206 includes display 1208 for showing information and inputs for user
The input equipment 1209 of such as mouse, keyboard etc of information.Wherein the display 1208 and input equipment 1209 all pass through
The input and output controller 1210 for being connected to system bus 1205 is connected to central processing unit 1201.The basic input/defeated
System 1206 can also include input and output controller 1210 to touch for receiving and handling from keyboard, mouse or electronics out
Control the input of multiple other equipment such as pen.Similarly, input and output controller 1210 also provide output to display screen, printer or
Other kinds of output equipment.
The mass-memory unit 1207 (is not shown by being connected to the bulk memory controller of system bus 1205
It is connected to central processing unit 1201 out).The mass-memory unit 1207 and its associated computer-readable medium are
Server 1200 provides non-volatile memories.That is, the mass-memory unit 1207 may include such as hard disk or
The computer-readable medium (not shown) of person's CD-ROM drive etc.
Without loss of generality, the computer-readable medium may include computer storage media and communication media.Computer
Storage medium includes information such as computer readable instructions, data structure, program module or other data for storage
The volatile and non-volatile of any method or technique realization, removable and irremovable medium.Computer storage medium includes
RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storages its technologies, CD-ROM, DVD or other optical storages, tape
Box, tape, disk storage or other magnetic storage devices.Certainly, skilled person will appreciate that the computer storage medium
It is not limited to above-mentioned several.Above-mentioned system storage 1204 and mass-memory unit 1207 may be collectively referred to as memory.
According to various embodiments of the present invention, the server 1200 can also be arrived by network connections such as internets
Remote computer operation on network.Namely server 1200 can be connect by the network being connected on the system bus 1205
Mouth unit 1211 is connected to network 1212, in other words, it is other kinds of to be connected to that Network Interface Unit 1211 also can be used
Network or remote computer system (not shown).
The memory further includes that one or more than one program, the one or more programs are stored in
In memory, and it is configured to be executed by one or more than one processor.Said one or more than one program include
For executing the instruction of the method for above-mentioned background server side.
In the exemplary embodiment, a kind of non-transitorycomputer readable storage medium including instruction, example are additionally provided
It such as include the memory of instruction, above-metioned instruction can be executed by the processor of terminal to complete sender visitor in above method embodiment
The each step or above-metioned instruction of family end or recipient's client-side are executed by the processor of server to complete the above method
Each step of background server side in embodiment.For example, the non-transitorycomputer readable storage medium can be ROM,
Random access memory (RAM), CD-ROM, tape, floppy disk and optical data storage devices etc..
It should be understood that referenced herein " multiple " refer to two or more."and/or", description association
The incidence relation of object indicates may exist three kinds of relationships, for example, A and/or B, can indicate: individualism A exists simultaneously A
And B, individualism B these three situations.Character "/" typicallys represent the relationship that forward-backward correlation object is a kind of "or".
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
Those of ordinary skill in the art will appreciate that realizing that all or part of the steps of above-described embodiment can pass through hardware
It completes, relevant hardware can also be instructed to complete by program, the program can store in a kind of computer-readable
In storage medium, storage medium mentioned above can be read-only memory, disk or CD etc..
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all in spirit of the invention and
Within principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.
Claims (10)
1. a kind of data processing method, which is characterized in that described method includes following steps:
Receive user's request that client is sent；
It is requested with user described in logged, and reads the log after the recording is completed；
The network transmitting-receiving of progress strong consistency in the case where the asynchronous remote of abstract network layer calls service framework (RPC)；
State machine engine (FSM) is called to receive strong consistency readjustment, so that the state machine engine executes user's request.
2. the method according to claim 1, wherein it is described receive client send user request include: by
Host node receives user's request from client, when receiving the request from client from node, refuses the request,
And the network address of the host node is returned to client.
3. the method according to claim 1, wherein being requested with user described in logged, and recording
Reading the log after completing includes:
Receive user's request；
User's request sequence is turned into Request Log；
The request of log rule is sent to disk read-write logic；
Reception of magnetic disc reads and writes the information that logic returns, and when rule is completed, reads the rule log.
4. the method according to claim 1, wherein the asynchronous remote in abstract network layer calls service framework
(RPC) network that strong consistency is carried out under, which is received and dispatched, includes:
The log of reading is synchronized to from node by host node；
It is described that rule operation is carried out to the log from node, and in success rule, notify the host node.
5. according to the method described in claim 4, it is characterized in that, the log of reading is synchronized to from node step by the host node
Suddenly include:
Host node sends heartbeat packet to from node；When not having to need synchronous log, the content of heartbeat packet is sky；It is same when needing
When walking log, then takes and need synchronous log；
Host node judges the timeout case from node to carry out disaster tolerance operation；
Judge whether host node is overtime from node according to the interval for receiving heartbeat packet；When host node time-out, then become waiting from node
Node is selected, and initiates election application.
6. the method according to claim 1, wherein the calling state machine engine (FSM) receives strong consistency
Readjustment, so that state machine engine execution user's request includes:
When host node receive it is most of when being successfully written journal file and notifying, called state machine engine (FSM) from node, and
It notifies the state machine engine (FSM) to carry out write operation to the log, then returns result to client.
7. the method according to claim 1, wherein after the state machine engine executes user request,
Implementing result is returned into the client.
8. the method according to claim 1, wherein the asynchronous remote in abstract network layer calls service framework
(RPC) during the network transmitting-receiving for carrying out strong consistency under, the parameter state for waiting single thread to return, when parameter state becomes
After the completion, other threads that asynchronous remote calls service framework (RPC) are continued to execute.
9. a kind of data processing equipment, which is characterized in that described method includes following steps:
Receiving module, for receiving user's request of client transmission；
Log read module for requesting with user described in logged, and reads the log after the recording is completed；
Strongly consistent transceiver module, for carrying out strong consistency in the case where the asynchronous remote of abstract network layer calls service framework (RPC)
Network transmitting-receiving；
Calling module, for calling state machine engine (FSM) to receive strong consistency readjustment, so that the state machine engine executes
User's request.
10. a kind of server, which is characterized in that the server includes device as claimed in claim 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811008654.3A CN109347906B (en) | 2018-08-30 | 2018-08-30 | Data transmission method, device and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811008654.3A CN109347906B (en) | 2018-08-30 | 2018-08-30 | Data transmission method, device and server |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109347906A true CN109347906A (en) | 2019-02-15 |
| CN109347906B CN109347906B (en) | 2021-04-20 |
Family
ID=65296682
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811008654.3A Active CN109347906B (en) | 2018-08-30 | 2018-08-30 | Data transmission method, device and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109347906B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110222118A (en) * | 2019-05-23 | 2019-09-10 | 上海易点时空网络有限公司 | Asynchronous data processing method and processing device based on queue |
| CN112671601A (en) * | 2020-12-11 | 2021-04-16 | 航天信息股份有限公司 | Zookeeper-based interface monitoring system and method |
| CN112769824A (en) * | 2021-01-07 | 2021-05-07 | 深圳市大富网络技术有限公司 | Information transmission state updating method, terminal, device and storage medium |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120166390A1 (en) * | 2010-12-23 | 2012-06-28 | Dwight Merriman | Method and apparatus for maintaining replica sets |
| CN103036961A (en) * | 2012-12-07 | 2013-04-10 | 蓝盾信息安全技术股份有限公司 | Distributed collection and storage method of journal |
| CN104283956A (en) * | 2014-09-30 | 2015-01-14 | 腾讯科技（深圳）有限公司 | Strong consistence distributed data storage method, device and system |
| CN104618127A (en) * | 2013-11-01 | 2015-05-13 | 深圳市腾讯计算机系统有限公司 | Master and standby storage node switching method and system |
| CN105122727A (en) * | 2013-01-11 | 2015-12-02 | Db网络公司 | Systems and methods for detecting and mitigating threats to a structured data storage system |
| US20160042043A1 (en) * | 2012-11-29 | 2016-02-11 | Red Hat, Inc. | Providing messages for a java message service |
| CN105512266A (en) * | 2015-12-03 | 2016-04-20 | 曙光信息产业(北京)有限公司 | Method and device for achieving operational consistency of distributed database |
| CN106201739A (en) * | 2016-06-29 | 2016-12-07 | 上海浦东发展银行股份有限公司信用卡中心 | A kind of remote invocation method of Storm based on Redis |
| CN106301938A (en) * | 2016-08-25 | 2017-01-04 | 成都索贝数码科技股份有限公司 | A kind of high availability and the data base cluster system of strong consistency and node administration method thereof |
| CN106789095A (en) * | 2017-03-30 | 2017-05-31 | 腾讯科技（深圳）有限公司 | Distributed system and message treatment method |
| US20170177277A1 (en) * | 2015-04-29 | 2017-06-22 | Internationl Business Machines Corporation | Managing data operations in a quorum-based data replication system |
-
2018
- 2018-08-30 CN CN201811008654.3A patent/CN109347906B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120166390A1 (en) * | 2010-12-23 | 2012-06-28 | Dwight Merriman | Method and apparatus for maintaining replica sets |
| US20160042043A1 (en) * | 2012-11-29 | 2016-02-11 | Red Hat, Inc. | Providing messages for a java message service |
| CN103036961A (en) * | 2012-12-07 | 2013-04-10 | 蓝盾信息安全技术股份有限公司 | Distributed collection and storage method of journal |
| CN105122727A (en) * | 2013-01-11 | 2015-12-02 | Db网络公司 | Systems and methods for detecting and mitigating threats to a structured data storage system |
| CN104618127A (en) * | 2013-11-01 | 2015-05-13 | 深圳市腾讯计算机系统有限公司 | Master and standby storage node switching method and system |
| CN104283956A (en) * | 2014-09-30 | 2015-01-14 | 腾讯科技（深圳）有限公司 | Strong consistence distributed data storage method, device and system |
| US20170177277A1 (en) * | 2015-04-29 | 2017-06-22 | Internationl Business Machines Corporation | Managing data operations in a quorum-based data replication system |
| CN105512266A (en) * | 2015-12-03 | 2016-04-20 | 曙光信息产业(北京)有限公司 | Method and device for achieving operational consistency of distributed database |
| CN106201739A (en) * | 2016-06-29 | 2016-12-07 | 上海浦东发展银行股份有限公司信用卡中心 | A kind of remote invocation method of Storm based on Redis |
| CN106301938A (en) * | 2016-08-25 | 2017-01-04 | 成都索贝数码科技股份有限公司 | A kind of high availability and the data base cluster system of strong consistency and node administration method thereof |
| CN106789095A (en) * | 2017-03-30 | 2017-05-31 | 腾讯科技（深圳）有限公司 | Distributed system and message treatment method |
Non-Patent Citations (1)
| Title |
|---|
| 佚名: ""数据复制与一致性"", 《HTTPS://WWW.CNBLOGS.COM/BITERROR/P/6909624.HTML》 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110222118A (en) * | 2019-05-23 | 2019-09-10 | 上海易点时空网络有限公司 | Asynchronous data processing method and processing device based on queue |
| CN110222118B (en) * | 2019-05-23 | 2022-04-05 | 上海易点时空网络有限公司 | Asynchronous data processing method and device based on queue |
| CN112671601A (en) * | 2020-12-11 | 2021-04-16 | 航天信息股份有限公司 | Zookeeper-based interface monitoring system and method |
| CN112671601B (en) * | 2020-12-11 | 2023-10-31 | 航天信息股份有限公司 | Interface monitoring system and method based on Zookeeper |
| CN112769824A (en) * | 2021-01-07 | 2021-05-07 | 深圳市大富网络技术有限公司 | Information transmission state updating method, terminal, device and storage medium |
| CN112769824B (en) * | 2021-01-07 | 2023-03-07 | 深圳市大富网络技术有限公司 | Information transmission state updating method, terminal, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109347906B (en) | 2021-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102571686B (en) | Method for realizing cloud conference system | |
| US20210042266A1 (en) | Geographically-distributed file system using coordinated namespace replication over a wide area network | |
| US9495381B2 (en) | Geographically-distributed file system using coordinated namespace replication over a wide area network | |
| RU2379755C2 (en) | System and method for sharing objects between computers over network | |
| KR101863398B1 (en) | Method and system for synchronization mechanism on multi-server reservation system | |
| US8719780B2 (en) | Application server with a protocol-neutral programming model for developing telecommunications-based applications | |
| CN110297801A (en) | A just transaction semantics for transaction system based on fault-tolerant FPGA | |
| US10193951B2 (en) | Binding CRUD-type protocols in distributed agreement protocols | |
| CN107832138B (en) | Method for realizing flattened high-availability namenode model | |
| US20030187927A1 (en) | Clustering infrastructure system and method | |
| CN104391930A (en) | Distributed file storage device and method | |
| CN109347906A (en) | A kind of data transmission method, device and server | |
| CN105337923B (en) | Data distributing method and system and data sending device and data sink | |
| TW201007489A (en) | Peer-to-peer redundant file server system and methods | |
| CN103207867A (en) | Method for processing data blocks, method for initiating recovery operation and nodes | |
| CN104011701A (en) | Content delivery network | |
| US10609123B2 (en) | Hybrid quorum policies for durable consensus in distributed systems | |
| CN101854351A (en) | The system and/or the method that are used for measured transmission of messages | |
| CN108710638A (en) | A kind of Distributed concurrency control method and system based on mixing RDMA operation | |
| CN112698838B (en) | Multi-cloud container deployment system and container deployment method thereof | |
| EP2318914B1 (en) | Method and apparatus for audit logging and role based security using one-way proxy architecture | |
| WO2021043124A1 (en) | Kbroker distributed operating system, storage medium, and electronic device | |
| CN114363154A (en) | Node election method and device, electronic equipment and storage medium | |
| WO2019100364A1 (en) | Dynamic resource allocation method in cloud video platform | |
| CN100486345C (en) | Business system based on PC server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |