CN111880947A - Data transmission method and device - Google Patents
Data transmission method and device Download PDFInfo
- Publication number
- CN111880947A CN111880947A CN202010697261.9A CN202010697261A CN111880947A CN 111880947 A CN111880947 A CN 111880947A CN 202010697261 A CN202010697261 A CN 202010697261A CN 111880947 A CN111880947 A CN 111880947A
- Authority
- CN
- China
- Prior art keywords
- data
- transmitted
- target client
- server
- sending
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000005540 biological transmission Effects 0.000 title claims abstract description 27
- 238000003860 storage Methods 0.000 claims abstract description 79
- 230000004044 response Effects 0.000 claims abstract description 74
- 230000002085 persistent effect Effects 0.000 claims abstract description 73
- 230000008439 repair process Effects 0.000 claims abstract description 9
- 238000009826 distribution Methods 0.000 claims description 35
- 238000012545 processing Methods 0.000 claims description 28
- 238000013507 mapping Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 9
- 238000004590 computer program Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000012986 modification Methods 0.000 description 5
- 230000004048 modification Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000002955 isolation Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0709—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/541—Client-server
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Information Transfer Between Computers (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a data transmission method and a device, wherein the method comprises the following steps: the server side obtains data to be transmitted from the data source side and stores the data to be transmitted into a cache space; if the server side determines that the target client side does not have a fault, the server side sends the data to be transmitted to the target client side; the server keeps the data to be transmitted stored in the cache space in a stage that the sending time length is less than the response waiting time length and the response message of the target client is not received; or if the server side determines that the target client side has a fault, storing the data to be transmitted in the cache space into a persistent storage space; and after the server side determines that the target client side is in fault repair, sending the data to be transmitted in the persistent storage space to the target client side.
Description
Technical Field
The present invention relates to the field of data synchronization, and in particular, to a data transmission method and apparatus.
Background
The architecture of conventional data transmission includes three parts: the system comprises a data source end, a server end and a client end. The data source end is used for generating data, the data need to be transmitted to a corresponding client end for specific logic processing, and the server end is responsible for receiving the data from the data source end and deciding how to distribute the data.
Currently, when a client fails, a server needs to temporarily stop distributing data to the client. After the client is recovered, the client needs to be reconfigured with the server, and data during the client repair period is supplemented. However, it is difficult to determine the specific time when the client fails, and data duplication and data loss may occur during the process of data backup. This is a problem to be solved.
Disclosure of Invention
The invention provides a data transmission method and a data transmission device, which solve the problems of data repetition and data loss possibly caused by data transmission in the prior art.
In a first aspect, the present invention provides a data transmission method, including: the server side obtains data to be transmitted from the data source side and stores the data to be transmitted into a cache space; if the server side determines that the target client side does not have a fault, the server side sends the data to be transmitted to the target client side; the server keeps the data to be transmitted stored in the cache space in a stage that the sending time length is less than the response waiting time length and the response message of the target client is not received; the target client is any one of a plurality of clients which are to consume the data to be transmitted; or if the server side determines that the target client side has a fault, storing the data to be transmitted in the cache space into a persistent storage space; and after the server side determines that the target client side is in fault repair, sending the data to be transmitted in the persistent storage space to the target client side.
In the method, the server firstly determines whether a target client fails or not, and sends the data to be transmitted to the target client when the target client fails, and before receiving the response message of the target client, keeping the data to be transmitted stored in the cache space, even if the target client fails before receiving the response message of the target client, the data to be transmitted is stored in the cache space and/or the persistent storage space, so that the data loss is not caused, and in addition, when the target client is determined to have a fault, the data to be transmitted in the cache space may be stored in the persistent storage space first, and after the target client is determined to be repaired, sending the data to be transmitted in the persistent storage space to the target client, so that the data is not sent repeatedly.
Optionally, if the server determines that the target client is not faulty, after sending the data to be transmitted to the target client, the method further includes: if the server side still does not receive the response message when the sending duration is greater than or equal to the response waiting duration, the server side determines that the target client side has a fault, and stores the data to be transmitted in the cache space into a persistent storage space; and after the server side determines that the target client side is in fault repair, sending the data to be transmitted in the persistent storage space to the target client side.
In the method, if the server does not receive the response message when the sending duration is greater than or equal to the response waiting duration, the server determines that the target client fails, and stores the data to be transmitted in the cache space into a persistent storage space, so that the target client is determined to fail in time based on the limited response waiting duration, the data to be transmitted in the cache space is stored into the persistent storage space in time, the space in the cache space is saved, and more cache resources are provided for the transmission data of the non-failed client.
Optionally, the server keeps the sending mode to the target client as a normal mode in a stage where it is not determined that the target client fails; the normal mode refers to that the server side sends data in the cache space to the target client side; further comprising: if the server side determines that the target client side has a fault, the sending mode is switched to a floor distribution mode; the floor distribution mode refers to that the server side sends data in the persistent storage space to the target client side.
In the method, the server keeps a sending mode to the target client as a normal mode in a stage that the target client is not determined to have a fault; the normal mode refers to that the server side sends data in the cache space to the target client side; further comprising: if the server side determines that the target client side has a fault, the sending mode is switched to a floor distribution mode; the floor distribution mode refers to that the server side sends data in the persistent storage space to the target client side.
Optionally, the server keeps the sending mode to the target client as a normal mode in a stage where it is not determined that the target client fails; the normal mode refers to that the server side sends data in the cache space to the target client side; further comprising: if the server side determines that the target client side has a fault, the sending mode is switched to a floor distribution mode; the floor distribution mode refers to that the server side sends data in the persistent storage space to the target client side.
In the above manner, the server keeps the sending mode to the target client as a normal mode in a stage that the target client is not determined to be in a fault, and sends data to the target client through the cache under the condition that the target client is not in the fault, so that data transmission is performed more rapidly in the normal mode, and the sending mode is switched to a floor distribution mode when the target client is in the fault, so that data repetition or data error between the server and the target client is further prevented.
Optionally, if the server determines that the target client fails, after the sending mode is switched to the floor distribution mode, the method further includes: and if the server side determines that all the data in the persistent storage space are successfully sent to the corresponding client side, the sending mode is switched back to the normal mode.
In the above manner, if the server determines that all the data in the persistent storage space are successfully sent to the corresponding client, the sending mode is switched back to the normal mode, so that the normal mode is recovered in time, and the efficiency of data transmission is improved.
Optionally, after the sending the data to be transmitted in the persistent storage space to the target client, the method further includes: and if the server side determines that the plurality of clients successfully receive the data to be transmitted, deleting the data to be transmitted from the persistent storage space.
In the above manner, after determining that the plurality of clients all successfully receive the data to be transmitted, the server does not need to transmit the data to be transmitted, so that the data to be transmitted is deleted from the persistent storage space in time, and the space of the persistent storage space is freed.
Optionally, the storing the data to be transmitted in the cache space into the persistent storage space includes: the server stores the data to be transmitted as a data file, and generates a file identifier of the data file according to a timestamp of the data to be transmitted and a data abstract of the data to be transmitted; and the server stores the data file and the mapping relation between the data file and the file identifier to the persistent storage space.
In the above manner, the server stores the data to be transmitted as a data file, generates a file identifier of the data file according to the timestamp of the data to be transmitted and the data summary of the data to be transmitted, and then persistently stores the file identifier, so that the data to be transmitted can be persistently tracked, and the traceability of the data to be transmitted is enhanced.
Optionally, if the server receives the response message within the time period that the sending time period is shorter than the response waiting time period, determining whether the response message indicates that the target client successfully responds; if successful response is indicated, the server side deletes the data to be transmitted from the cache space; and if the unsuccessful response is indicated, the server resends the data to be transmitted.
In the above manner, after receiving the response message, if a successful response is indicated, the server deletes the data to be transmitted from the cache space in time, and if an unsuccessful response is indicated, the server resends the data to be transmitted, so that the cache space can be vacated in time when the data is successfully transmitted, and the data can be retransmitted in time when the data is damaged, thereby ensuring the correctness of data transmission.
In a second aspect, the present invention provides a data transmission apparatus, including: the acquisition module is used for acquiring data to be transmitted from a data source end and storing the data to be transmitted into a cache space; the processing module is used for sending the data to be transmitted to the target client if the target client is determined not to have a fault; keeping the data to be transmitted stored in the cache space in a stage that the sending duration is less than the response waiting duration and the response message of the target client is not received; the target client is any one of a plurality of clients which are to consume the data to be transmitted; or if the target client is determined to be in fault, storing the data to be transmitted in the cache space into a persistent storage space; and after the target client is determined to be repaired, sending the data to be transmitted in the persistent storage space to the target client.
Optionally, the processing module is further configured to: if the response message is not received when the sending duration is longer than or equal to the response waiting duration, determining that the target client fails, and storing the data to be transmitted in the cache space into a persistent storage space; and after the target client is determined to be repaired, sending the data to be transmitted in the persistent storage space to the target client.
Optionally, the processing module is further configured to: keeping the sending mode of the target client as a normal mode in a stage that the target client is not determined to have a fault; the processing module is specifically configured to: if the target client is determined to be in fault, switching the sending mode into a ground distribution mode; the floor distribution mode refers to sending data in the persistent storage space to the target client.
Optionally, the processing module is further configured to: and if the data in the persistent storage space are determined to be all successfully sent to the corresponding client, switching the sending mode back to the normal mode.
Optionally, the processing module is further configured to: and if the plurality of clients are determined to successfully receive the data to be transmitted, deleting the data to be transmitted from the persistent storage space.
Optionally, the processing module is specifically configured to: storing the data to be transmitted as a data file, and generating a file identifier of the data file according to the timestamp of the data to be transmitted and the data abstract of the data to be transmitted; and storing the data file and the mapping relation between the data file and the file identifier to the persistent storage space.
Optionally, the processing module is further configured to: if the response message is received within the time length that the sending time length is shorter than the response waiting time length, determining whether the response message indicates that the target client successfully responds; if successful response is indicated, deleting the data to be transmitted from the cache space; and if the unsuccessful response is indicated, retransmitting the data to be transmitted.
The advantageous effects of the second aspect and the various optional apparatuses of the second aspect may refer to the advantageous effects of the first aspect and the various optional methods of the first aspect, and are not described herein again.
In a third aspect, the present invention provides a computer device comprising a program or instructions for performing the method of the first aspect and the alternatives of the first aspect when the program or instructions are executed.
In a fourth aspect, the present invention provides a storage medium comprising a program or instructions which, when executed, is adapted to perform the method of the first aspect and the alternatives of the first aspect.
These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described hereinafter.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a schematic flowchart of a data transmission method according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of a data transmission method according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of a data transmission device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The architecture of conventional data transmission includes three parts: the system comprises a data source end, a server end and a client end. In a data distribution system and a main stream message middleware system, a server needs to process a client fault when the client fails and take a countermeasure to ensure that the data distribution system can normally and stably operate and ensure the integrity of distributed data. However, it is difficult to determine the specific time when the client fails, and data duplication and data loss may occur during the process of data backup. This is a problem to be solved. Therefore, as shown in fig. 1, a data transmission method is provided in the embodiments of the present application.
Step 101: the server side obtains data to be transmitted from the data source side and stores the data to be transmitted into the cache space.
Step 102: if the server side determines that the target client side does not have a fault, the server side sends the data to be transmitted to the target client side; and the server keeps the data to be transmitted stored in the cache space in a stage that the sending time length is less than the response waiting time length and the response message of the target client is not received.
The target client is any one of a plurality of clients which wait to consume the data to be transmitted.
Step 103: if the server side determines that the target client side has a fault, the server side stores the data to be transmitted in the cache space into a persistent storage space; and after the server side determines that the target client side is in fault repair, sending the data to be transmitted in the persistent storage space to the target client side.
The persistent storage space may have various spaces such as a disk space and a hard disk space.
It should be noted that step 102 and step 103 are not in a sequential execution relationship with each other, and step 102 and step 103 may be executed in parallel.
In step 102, it may be determined whether the target client fails through a timeout mechanism. For example, if the response message is received when the response waiting time is not exceeded, it is determined that the target client has not failed. And whether the target client fails or not can be determined by reading the running state of the target client, keeping a heartbeat test with the target client and the like. For example, the response waiting time period is set to 10 seconds. If the server receives the response message, the server determines that the target client does not have a fault. For example, the sending time is within 0-10 seconds, such as at the end of 7 seconds, and the data to be transmitted is kept stored in the buffer space within 0-7 seconds. Or if the sending time length exceeds 10 seconds and the response message is still not received, keeping the data to be transmitted stored in the cache space within the first 0-10 seconds.
In step 103, if it is determined that the target client is faulty, the server first stores the data to be transmitted in the cache space into a persistent storage space, and then sends the data to be transmitted in the persistent storage space to the target client, and sends the data to be transmitted to the target client, so that strict integrity of the data can be ensured. Moreover, for other clients, the data to be transmitted in the persistent storage space can be sent to other clients, and the influence between the clients under the condition of one-to-many can be reduced.
The scheme has high availability, can ensure that one data is not lost when a fault occurs, ensures the integrity of the distributed data, has automatic and unsupervised functions, can automatically complete the isolation and recovery operation of the fault client, does not need manual intervention, has decoupling performance, and can reduce the influence between the clients of the data distribution system under the one-to-many condition.
After step 102 is performed, an alternative embodiment is as follows: if the server side still does not receive the response message when the sending duration is greater than or equal to the response waiting duration, the server side determines that the target client side has a fault, and stores the data to be transmitted in the cache space into a persistent storage space; and after the server side determines that the target client side is in fault repair, sending the data to be transmitted in the persistent storage space to the target client side.
And if the sending time length exceeds 10 seconds and the response message is still not received, the server determines that the target client side has a fault. And then, the data to be transmitted in the cache space is still stored in a persistent storage space, and after the server determines that the target client is repaired due to the fault, the data to be transmitted in the persistent storage space is sent to the target client.
The method of step 101 to step 103 may be configured to have a transmission mode. In an optional implementation manner (hereinafter, referred to as an optional implementation manner of the sending mode), the server keeps the sending mode to the target client as a normal mode in a stage where it is not determined that the target client fails; if the server side determines that the target client side has a fault, the sending mode is switched to a floor distribution mode; the floor distribution mode refers to that the server side sends data in the persistent storage space to the target client side.
That is, in the normal mode, the server sends the data in the cache space to the target client, and in the floor distribution mode, the server sends the data in the persistent storage space to the target client.
Specifically, the method can achieve the effect of automatic isolation optimization, when a target client is abnormal, the server can automatically switch to a floor distribution mode, data in the cache space is written into the persistent storage space, and meanwhile, a trigger monitoring alarm mechanism can be used for reminding the target client of a fault waiting for repair.
An alternative implementation based on the transmission mode is as follows:
and if the server side determines that all the data in the persistent storage space are successfully sent to the corresponding client side, the sending mode is switched back to the normal mode.
Specifically, after the target client is recovered, the server reads data from the persistent storage space and sends the data to the client, and the server automatically switches back to the normal mode until all data in the persistent storage space are completely distributed.
An alternative implementation of step 103 is as follows:
the server stores the data to be transmitted as a data file, and generates a file identifier of the data file according to a timestamp of the data to be transmitted and a data abstract of the data to be transmitted; and the server stores the data file and the mapping relation between the data file and the file identifier to the persistent storage space.
The data file generates the file identifier of the data file according to the timestamp of the data to be transmitted and the data abstract of the data to be transmitted, and stores the mapping relation between the data file and the file identifier, so that the traceability is realized, and the development of other functions can be realized.
An alternative implementation after step 103 is performed is as follows:
and if the server side determines that the plurality of clients successfully receive the data to be transmitted, deleting the data to be transmitted from the persistent storage space.
It should be noted that, in the above optional embodiment, all the clients need to successfully receive the data to be transmitted, so that it is ensured that the data to be transmitted does not need to be generated, and the persistent storage space is saved.
An alternative implementation of steps 101 to 103 is as follows:
if the server receives the response message within the time length of the sending time length smaller than the response waiting time length, the server determines whether the response message indicates that the target client successfully responds; if successful response is indicated, the server side deletes the data to be transmitted from the cache space; and if the unsuccessful response is indicated, the server resends the data to be transmitted.
Specifically, a normal sending module in a normal mode, a landing distribution module in a landing distribution mode, and a landing distribution response module may be set in the server, and the landing distribution module is called when the server sends data. After receiving the call of the server, the floor distribution module calls the floor distribution module again for retry through the data distribution system from the persistent storage space, and the specific retry process can be realized through a general timeout mechanism.
And the ground distribution response module receives the response message returned by the target client, marks that the data to be transmitted is successfully sent to the target task if successful response is indicated, and moves the data to be transmitted in the cache space out to a backup catalog for subsequent verification. If the indication is unsuccessful, the server directly reports the error and discards the response, waits for the general overtime mechanism of the server to recall the normal sending module and sends the response again. If successful response is indicated, the server side deletes the data to be transmitted from the cache space; and if the unsuccessful response is indicated, the server resends the data to be transmitted.
Combining the method shown in fig. 1 and the optional method, further, the overall design flow of the present solution can be as shown in fig. 2.
a) In the normal mode, the server receives data sent by the data source end, stores the data in a designated cache block (for example, a designated partition in Kafka) in a cache space, judges whether the state of the target client is normal, and if the state is normal, normally distributes the data and finishes exiting; otherwise, continuing the b processing.
b) And switching the distribution mode of the server side into a ground distribution mode, writing the data transmitted from the data source side received later into the persistent storage space, and continuing the c processing.
c) Updating a file identifier of a data file of data to be transmitted in the persistent storage space, then calling a floor distribution module to transmit the data file, judging whether the calling of the transmission module is overtime, and calling the transmission module again to retransmit if the calling of the transmission module is overtime; otherwise, continuing the d processing.
d) Calling a floor distribution response module, judging whether the response module is successfully called, and if the response module is failed, performing c processing; otherwise, entering e processing.
e) And judging whether unsent data exists in the persistent storage space, and if so, executing c processing on the unsent data. If not, judging that the data is completely supplemented when the consumer fails, recovering the state of the server side to be in a normal distribution mode, and ending the exit.
As shown in fig. 3, the present invention provides a data transmission apparatus, including: an obtaining module 301, configured to obtain data to be transmitted from a data source, and store the data to be transmitted in a cache space; a processing module 302, configured to send the data to be transmitted to a target client if it is determined that the target client does not have a fault; keeping the data to be transmitted stored in the cache space in a stage that the sending duration is less than the response waiting duration and the response message of the target client is not received; the target client is any one of a plurality of clients which are to consume the data to be transmitted; or if the target client is determined to be in fault, storing the data to be transmitted in the cache space into a persistent storage space; and after the target client is determined to be repaired, sending the data to be transmitted in the persistent storage space to the target client.
Optionally, the processing module 302 is further configured to: if the response message is not received when the sending duration is longer than or equal to the response waiting duration, determining that the target client fails, and storing the data to be transmitted in the cache space into a persistent storage space; and after the target client is determined to be repaired, sending the data to be transmitted in the persistent storage space to the target client.
Optionally, the processing module 302 is further configured to: keeping the sending mode of the target client as a normal mode in a stage that the target client is not determined to have a fault; the processing module 302 is specifically configured to: if the target client is determined to be in fault, switching the sending mode into a ground distribution mode; the floor distribution mode refers to sending data in the persistent storage space to the target client.
Optionally, the processing module 302 is further configured to: and if the data in the persistent storage space are determined to be all successfully sent to the corresponding client, switching the sending mode back to the normal mode.
Optionally, the processing module 302 is further configured to: and if the plurality of clients are determined to successfully receive the data to be transmitted, deleting the data to be transmitted from the persistent storage space.
Optionally, the processing module 302 is specifically configured to: storing the data to be transmitted as a data file, and generating a file identifier of the data file according to the timestamp of the data to be transmitted and the data abstract of the data to be transmitted; and storing the data file and the mapping relation between the data file and the file identifier to the persistent storage space.
Optionally, the processing module 302 is further configured to: if the response message is received within the time length that the sending time length is shorter than the response waiting time length, determining whether the response message indicates that the target client successfully responds; if successful response is indicated, deleting the data to be transmitted from the cache space; and if the unsuccessful response is indicated, retransmitting the data to be transmitted.
Based on the same inventive concept, embodiments of the present invention also provide a computer device, which includes a program or instructions, and when the program or instructions are executed by the computer device, the data processing method and any optional method shown in fig. 1 are executed.
Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, which includes a program or instructions, and when the program or instructions are run on a computer, the program or instructions cause the computer to execute the data processing method and any optional method shown in fig. 1.
It should be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (10)
1. A method of data transmission, comprising:
the server side obtains data to be transmitted from the data source side and stores the data to be transmitted into a cache space;
if the server side determines that the target client side does not have a fault, the server side sends the data to be transmitted to the target client side; the server keeps the data to be transmitted stored in the cache space in a stage that the sending time length is less than the response waiting time length and the response message of the target client is not received; the target client is any one of a plurality of clients which are to consume the data to be transmitted; alternatively, the first and second electrodes may be,
if the server side determines that the target client side has a fault, the server side stores the data to be transmitted in the cache space into a persistent storage space; and after the server side determines that the target client side is in fault repair, sending the data to be transmitted in the persistent storage space to the target client side.
2. The method of claim 1, wherein if the server determines that the target client is not faulty, then after sending the data to be transmitted to the target client, the method further comprises:
if the server side still does not receive the response message when the sending duration is greater than or equal to the response waiting duration, the server side determines that the target client side has a fault, and stores the data to be transmitted in the cache space into a persistent storage space; and after the server side determines that the target client side is in fault repair, sending the data to be transmitted in the persistent storage space to the target client side.
3. The method of claim 1, wherein the server maintains a sending mode to the target client as a normal mode in a stage where it is not determined that the target client fails; the normal mode refers to that the server side sends data in the cache space to the target client side; further comprising:
if the server side determines that the target client side has a fault, the sending mode is switched to a floor distribution mode; the floor distribution mode refers to that the server side sends data in the persistent storage space to the target client side.
4. The method of claim 3, wherein if the server determines that the target client fails, after switching the sending mode to a floor distribution mode, the method further comprises:
and if the server side determines that all the data in the persistent storage space are successfully sent to the corresponding client side, the sending mode is switched back to the normal mode.
5. The method of any of claims 1 to 4, wherein after sending the data to be transmitted in the persistent storage space to the target client, further comprising:
and if the server side determines that the plurality of clients successfully receive the data to be transmitted, deleting the data to be transmitted from the persistent storage space.
6. The method of any of claims 1 to 4, wherein storing the data to be transmitted in the cache space in the persistent storage space comprises:
the server stores the data to be transmitted as a data file, and generates a file identifier of the data file according to a timestamp of the data to be transmitted and a data abstract of the data to be transmitted;
and the server stores the data file and the mapping relation between the data file and the file identifier to the persistent storage space.
7. The method of any of claims 1 to 4, further comprising:
if the server receives the response message within the time length of the sending time length smaller than the response waiting time length, the server determines whether the response message indicates that the target client successfully responds;
if successful response is indicated, the server side deletes the data to be transmitted from the cache space; and if the unsuccessful response is indicated, the server resends the data to be transmitted.
8. A data transmission apparatus, comprising:
the acquisition module is used for acquiring data to be transmitted from a data source end and storing the data to be transmitted into a cache space;
the processing module is used for sending the data to be transmitted to the target client if the target client is determined not to have a fault; keeping the data to be transmitted stored in the cache space in a stage that the sending duration is less than the response waiting duration and the response message of the target client is not received; the target client is any one of a plurality of clients which are to consume the data to be transmitted; alternatively, the first and second electrodes may be,
if the target client is determined to be in fault, storing the data to be transmitted in the cache space into a persistent storage space; and after the target client is determined to be repaired, sending the data to be transmitted in the persistent storage space to the target client.
9. A computer device comprising a program or instructions that, when executed, perform the method of any of claims 1 to 7.
10. A computer-readable storage medium comprising a program or instructions which, when executed, perform the method of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010697261.9A CN111880947B (en) | 2020-07-20 | 2020-07-20 | Data transmission method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010697261.9A CN111880947B (en) | 2020-07-20 | 2020-07-20 | Data transmission method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111880947A true CN111880947A (en) | 2020-11-03 |
| CN111880947B CN111880947B (en) | 2023-12-15 |
Family
ID=73155845
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010697261.9A Active CN111880947B (en) | 2020-07-20 | 2020-07-20 | Data transmission method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111880947B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113034283A (en) * | 2021-03-29 | 2021-06-25 | 中国银联股份有限公司 | Transaction data processing method, device and system |
| CN113783939A (en) * | 2021-08-20 | 2021-12-10 | 奇安信科技集团股份有限公司 | File transmission method and device, electronic equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105933380A (en) * | 2016-04-01 | 2016-09-07 | 宇龙计算机通信科技(深圳)有限公司 | Data transmission method, device and terminal |
| US20170329740A1 (en) * | 2016-05-16 | 2017-11-16 | Catherine H. Crawford | Distributed client based cache for keys using demand fault invalidation |
| CN110505275A (en) * | 2019-07-16 | 2019-11-26 | 北京奇艺世纪科技有限公司 | Data transmission method, system, server and computer readable storage medium |
| US20200042444A1 (en) * | 2018-08-02 | 2020-02-06 | MemVerge, Inc | Distributed Memory Object Architecture |
| CN110795166A (en) * | 2019-10-28 | 2020-02-14 | 深圳前海微众银行股份有限公司 | Data processing method and device |
-
2020
- 2020-07-20 CN CN202010697261.9A patent/CN111880947B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105933380A (en) * | 2016-04-01 | 2016-09-07 | 宇龙计算机通信科技(深圳)有限公司 | Data transmission method, device and terminal |
| US20170329740A1 (en) * | 2016-05-16 | 2017-11-16 | Catherine H. Crawford | Distributed client based cache for keys using demand fault invalidation |
| US20200042444A1 (en) * | 2018-08-02 | 2020-02-06 | MemVerge, Inc | Distributed Memory Object Architecture |
| CN110505275A (en) * | 2019-07-16 | 2019-11-26 | 北京奇艺世纪科技有限公司 | Data transmission method, system, server and computer readable storage medium |
| CN110795166A (en) * | 2019-10-28 | 2020-02-14 | 深圳前海微众银行股份有限公司 | Data processing method and device |
Non-Patent Citations (1)
| Title |
|---|
| 张沪滨等: "面向QEMU的分布式块存储系统的设计与实现", 《微型电脑应用》, vol. 32, no. 04, pages 54 - 57 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113034283A (en) * | 2021-03-29 | 2021-06-25 | 中国银联股份有限公司 | Transaction data processing method, device and system |
| CN113783939A (en) * | 2021-08-20 | 2021-12-10 | 奇安信科技集团股份有限公司 | File transmission method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111880947B (en) | 2023-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108847982B (en) | Distributed storage cluster and node fault switching method and device thereof | |
| US5590277A (en) | Progressive retry method and apparatus for software failure recovery in multi-process message-passing applications | |
| US5440726A (en) | Progressive retry method and apparatus having reusable software modules for software failure recovery in multi-process message-passing applications | |
| CN105933407B (en) | method and system for realizing high availability of Redis cluster | |
| US5530802A (en) | Input sequence reordering method for software failure recovery | |
| JP3253883B2 (en) | Process restart method and process monitoring device | |
| US7856639B2 (en) | Monitoring and controlling applications executing in a computing node | |
| CN111782360A (en) | Distributed task scheduling method and device | |
| EP3142011A1 (en) | Anomaly recovery method for virtual machine in distributed environment | |
| CN110807064B (en) | Data recovery device in RAC distributed database cluster system | |
| US20120011100A1 (en) | Snapshot acquisition processing technique | |
| US9430221B2 (en) | Device and method for updating firmware of a rackmount server system | |
| CN111064789B (en) | Data migration method and system | |
| CN111880947B (en) | Data transmission method and device | |
| CN102833281A (en) | Method, device and system for realizing distributed automatically-increasing counting | |
| CN106815094B (en) | Method and equipment for realizing transaction submission in master-slave synchronization mode | |
| CN111858077A (en) | Recording method, device and equipment for IO request log in storage system | |
| CN107943567B (en) | High-reliability task scheduling method and system based on AMQP protocol | |
| EP4060514A1 (en) | Distributed database system and data disaster backup drilling method | |
| CN116185697B (en) | Container cluster management method, device and system, electronic equipment and storage medium | |
| CN111371582B (en) | PCIE link fault processing method and device | |
| CN105373549A (en) | Data migration method and device and data node server | |
| CN115599310B (en) | Method and device for controlling storage resources in storage node and storage node | |
| CN116991710B (en) | Automatic test method and system, electronic equipment and storage medium thereof | |
| WO2024113534A1 (en) | Method and apparatus for controlling storage resources in storage node, and storage node |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |