CN111880947B - Data transmission method and device - Google Patents
Data transmission method and device Download PDFInfo
- Publication number
- CN111880947B CN111880947B CN202010697261.9A CN202010697261A CN111880947B CN 111880947 B CN111880947 B CN 111880947B CN 202010697261 A CN202010697261 A CN 202010697261A CN 111880947 B CN111880947 B CN 111880947B
- Authority
- CN
- China
- Prior art keywords
- data
- transmitted
- target client
- server
- sending
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 230000005540 biological transmission Effects 0.000 title claims abstract description 35
- 238000003860 storage Methods 0.000 claims abstract description 81
- 230000002085 persistent effect Effects 0.000 claims abstract description 76
- 230000004044 response Effects 0.000 claims abstract description 73
- 238000009826 distribution Methods 0.000 claims description 33
- 238000012545 processing Methods 0.000 claims description 25
- 230000008439 repair process Effects 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000007246 mechanism Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 230000004075 alteration Effects 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 230000001502 supplementing effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0709—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0793—Remedial or corrective actions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/541—Client-server
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Information Transfer Between Computers (AREA)
- Computer And Data Communications (AREA)
Abstract
The application discloses a data transmission method and a device, wherein the method comprises the following steps: the method comprises the steps that a server side obtains data to be transmitted from a data source side and stores the data to be transmitted into a cache space; if the server determines that the target client does not fail, the server sends the data to be transmitted to the target client; the server keeps the data to be transmitted stored in the cache space in a stage that the sending duration is smaller than the response waiting duration and the response message of the target client is not received; or if the server determines that the target client fails, storing the data to be transmitted in the cache space into a persistent storage space; and after the server determines that the target client has fault repaired, sending the data to be transmitted in the persistent storage space to the target client.
Description
Technical Field
The present application relates to the field of data synchronization, and in particular, to a data transmission method and apparatus.
Background
The architecture of conventional data transmission includes three parts: the system comprises a data source end, a service end and a client. The data source end is used for generating data, the data need to be transmitted to the corresponding client end to carry out specific logic processing, and the server end is responsible for receiving the data from the data source end and deciding how to distribute the data.
Currently, when a client fails, the server needs to temporarily stop distributing data to the client. After the client recovers, the client needs to reconfigure with the server and make up for the data during the client repair. However, it is difficult to determine the specific time of the failure of the client, and in the process of supplementing data, data repetition and data loss may be caused. This is a problem to be solved.
Disclosure of Invention
The application provides a data transmission method and device, which solve the problems that data transmission possibly causes data repetition and data loss in the prior art.
In a first aspect, the present application provides a data transmission method, including: the method comprises the steps that a server side obtains data to be transmitted from a data source side and stores the data to be transmitted into a cache space; if the server determines that the target client does not fail, the server sends the data to be transmitted to the target client; the server keeps the data to be transmitted stored in the cache space in a stage that the sending duration is smaller than the response waiting duration and the response message of the target client is not received; the target client is any one of a plurality of clients which are to consume the data to be transmitted; or if the server determines that the target client fails, storing the data to be transmitted in the cache space into a persistent storage space; and after the server determines that the target client has fault repaired, sending the data to be transmitted in the persistent storage space to the target client.
In the method, the server side determines whether the target client side fails or not, the data to be transmitted is sent to the target client side when the target client side fails, and keeps the data to be transmitted stored in the cache space before receiving the response message of the target client side, even if the target client side fails before receiving the response message of the target client side, the data to be transmitted is stored in the cache space and/or the persistent storage space, so that data loss cannot be caused, in addition, when the target client side fails, the data to be transmitted in the cache space can be stored in the persistent storage space, and after the target client side is determined to fail, the data to be transmitted in the persistent storage space is sent to the target client side, so that data cannot be repeatedly sent.
Optionally, if the server determines that the target client fails, after sending the data to be transmitted to the target client, the method further includes: if the server side still does not receive the response message when the sending time is longer than or equal to the response waiting time, the server side determines that the target client side fails and stores the data to be transmitted in the cache space into a persistent storage space; and after the server determines that the target client has fault repaired, sending the data to be transmitted in the persistent storage space to the target client.
In the method, if the server side still does not receive the response message when the sending time length is longer than or equal to the response waiting time length, the server side determines that the target client side fails, and stores the data to be transmitted in the cache space into the persistent storage space, so that the target client side fails based on the defined response waiting time length, the data to be transmitted in the cache space is stored into the persistent storage space in time, the space in the cache space is saved, and more cache resources are provided for the transmission data of the non-failed client side.
Optionally, the server maintains the sending mode of the target client as a normal mode in a stage where the failure of the target client is not determined; the normal mode refers to that the server side sends data in the cache space to the target client side; further comprises: if the server determines that the target client fails, the sending mode is switched to a floor distribution mode; the floor distribution mode refers to that the server side sends data in the persistent storage space to the target client side.
In the method, the server maintains the sending mode of the target client as a normal mode in a stage of not determining that the target client fails; the normal mode refers to that the server side sends data in the cache space to the target client side; further comprises: if the server determines that the target client fails, the sending mode is switched to a floor distribution mode; the floor distribution mode refers to that the server side sends data in the persistent storage space to the target client side.
Optionally, the server maintains the sending mode of the target client as a normal mode in a stage where the failure of the target client is not determined; the normal mode refers to that the server side sends data in the cache space to the target client side; further comprises: if the server determines that the target client fails, the sending mode is switched to a floor distribution mode; the floor distribution mode refers to that the server side sends data in the persistent storage space to the target client side.
In the above manner, the server maintains the sending mode of the target client as the normal mode in the stage that the target client is not determined to be faulty, and when the target client is not faulty, the data is sent to the target client through the buffer memory, and the data is more rapidly transmitted in the normal mode, and when the fault occurs, the sending mode is switched to the floor distribution mode, so that the data of the server and the data in the target client are further prevented from being repeated or data errors.
Optionally, if the server determines that the target client fails, after switching the sending mode to the floor distribution mode, the method further includes: and if the server determines that all the data in the persistent storage space are successfully transmitted to the corresponding client, switching the transmission mode back to the normal mode.
In the above manner, if the server determines that all the data in the persistent storage space are successfully transmitted to the corresponding client, the transmitting mode is switched back to the normal mode, so that the normal mode is timely recovered, and the efficiency of data transmission is improved.
Optionally, after the sending the data to be transmitted in the persistent storage space to the target client, the method further includes: and if the server side determines that the plurality of clients successfully receive the data to be transmitted, deleting the data to be transmitted from the persistent storage space.
In the above manner, after determining that the plurality of clients successfully receive the data to be transmitted, the server does not need to transmit the data to be transmitted, so that the data to be transmitted is deleted from the persistent storage space in time, and the space of the persistent storage space is vacated.
Optionally, the storing the data to be transmitted in the buffer space in the persistent storage space includes: the server stores the data to be transmitted into a data file, and generates a file identification of the data file according to the timestamp of the data to be transmitted and the data abstract of the data to be transmitted; and the server stores the data file and the mapping relation between the data file and the file identifier into the persistent storage space.
In the above manner, the server stores the data to be transmitted as a data file, generates the file identifier of the data file according to the timestamp of the data to be transmitted and the data abstract of the data to be transmitted, and then performs persistent storage, so that the data to be transmitted can be tracked in a persistent manner, and the traceability of the data to be transmitted is enhanced.
Optionally, if the sending duration is less than the response waiting duration, the server side receives the response message, and determines whether the response message indicates that the target client side successfully responds; if the successful response is indicated, the server deletes the data to be transmitted from the cache space; and if the response is not successful, the server resends the data to be transmitted.
In the above manner, after receiving the response message, the server timely deletes the data to be transmitted from the buffer space if a successful response is indicated, and resends the data to be transmitted if a unsuccessful response is indicated, so that the buffer space can be emptied in time when the data transmission is successful, and the data can be retransmitted in time when the data is damaged, thereby ensuring the accuracy of data transmission.
In a second aspect, the present application provides a data transmission apparatus comprising: the acquisition module is used for acquiring data to be transmitted from a data source end and storing the data to be transmitted into a cache space; the processing module is used for sending the data to be transmitted to the target client if the target client is determined to not have a fault; in the stage that the sending duration is less than the response waiting duration and the response message of the target client is not received, keeping the data to be transmitted stored in the cache space; the target client is any one of a plurality of clients which are to consume the data to be transmitted; or if the target client fails, storing the data to be transmitted in the cache space into a persistent storage space; and after the fault repair of the target client is determined, sending the data to be transmitted in the persistent storage space to the target client.
Optionally, the processing module is further configured to: if the response message is not received when the sending time length is longer than or equal to the response waiting time length, determining that the target client fails, and storing the data to be transmitted in the cache space into a persistent storage space; and after the fault repair of the target client is determined, sending the data to be transmitted in the persistent storage space to the target client.
Optionally, the processing module is further configured to: maintaining a normal mode of a transmission mode to the target client in a stage in which the target client is not determined to be faulty; the processing module is specifically configured to: if the target client side is determined to have a fault, switching the sending mode to a floor distribution mode; the floor distribution mode refers to sending data in the persistent storage space to the target client.
Optionally, the processing module is further configured to: and if the data in the persistent storage space are determined to be successfully transmitted to the corresponding client, switching the transmission mode back to the normal mode.
Optionally, the processing module is further configured to: and if the plurality of clients are determined to successfully receive the data to be transmitted, deleting the data to be transmitted from the persistent storage space.
Optionally, the processing module is specifically configured to: storing the data to be transmitted into a data file, and generating a file identification of the data file according to the timestamp of the data to be transmitted and the data abstract of the data to be transmitted; and storing the data file and the mapping relation between the data file and the file identifier into the persistent storage space.
Optionally, the processing module is further configured to: if the sending duration is smaller than the response waiting duration, the response message is received, and whether the response message indicates the target client to successfully respond or not is determined; if the successful response is indicated, deleting the data to be transmitted from the cache space; and if the response is not successful, retransmitting the data to be transmitted.
The advantages of the foregoing second aspect and the advantages of the foregoing optional apparatuses of the second aspect may refer to the advantages of the foregoing first aspect and the advantages of the foregoing optional methods of the first aspect, and will not be described herein.
In a third aspect, the present application provides a computer device comprising a program or instructions which, when executed, is operable to perform the above-described first aspect and the respective alternative methods of the first aspect.
In a fourth aspect, the present application provides a storage medium comprising a program or instructions which, when executed, is adapted to carry out the above-described first aspect and the respective alternative methods of the first aspect.
These and other aspects of the application will be more readily apparent from the following description of the embodiments.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it will be apparent that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flow chart corresponding to a data transmission method according to an embodiment of the present application;
fig. 2 is a schematic flowchart corresponding to a data transmission method according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a data transmission device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be described in further detail below with reference to the accompanying drawings, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
The architecture of conventional data transmission includes three parts: the system comprises a data source end, a service end and a client. In the data distribution system and the main stream message middleware system, the server side needs to process the client side faults when the client side faults, and make countermeasures, so that the data distribution system can normally and stably operate, and the integrity of distributed data is ensured. However, it is difficult to determine the specific time of the failure of the client, and in the process of supplementing data, data repetition and data loss may be caused. This is a problem to be solved. For this reason, as shown in fig. 1, a data transmission method is provided in an embodiment of the present application.
Step 101: the server side obtains data to be transmitted from the data source side and stores the data to be transmitted into the cache space.
Step 102: if the server determines that the target client does not fail, the server sends the data to be transmitted to the target client; and the server keeps the data to be transmitted stored in the cache space in a stage that the sending duration is less than the response waiting duration and the response message of the target client is not received.
The target client is any one of a plurality of clients which are to consume the data to be transmitted.
Step 103: if the server determines that the target client fails, storing the data to be transmitted in the cache space into a persistent storage space; and after the server determines that the target client has fault repaired, sending the data to be transmitted in the persistent storage space to the target client.
The persistent storage space can be various types of disk space and hard disk space.
It should be noted that, the step 102 and the step 103 are not constrained to be executed in sequence, and the step 102 and the step 103 may be executed in parallel.
In step 102, it may be determined whether the target client has failed through a timeout mechanism. For example, if the response message is received when the response waiting time is not exceeded, it is determined that the target client is not malfunctioning. And determining whether the target client fails or not by reading the running state of the target client, maintaining a heartbeat test with the target client and the like. For example, the answer waiting period is set to 10 seconds. And if the server receives the response message, the server determines that the target client does not fail. For example, the sending duration is within 0-10 seconds, such as the end of 7 seconds, and then the data to be transmitted is kept stored in the buffer space within 0-7 seconds. Or if the sending time exceeds 10 seconds and the response message is still not received, keeping the data to be transmitted stored in the buffer space in the first 0-10 seconds.
In step 103, if the server determines that the target client fails, the server stores the data to be transmitted in the buffer space into a persistent storage space, and then sends the data to be transmitted in the persistent storage space to the target client, so as to ensure strict integrity of the data. Moreover, the data to be transmitted in the persistent storage space can be sent to other clients, so that the influence among the clients in the one-to-many condition can be reduced.
The scheme has high availability, can ensure that one data is not lost when a fault occurs, ensures the integrity of distributed data, has an automatic and unsupervised function, can automatically complete the isolation and recovery operation of a fault client, does not need manual intervention, has decoupling property, and can reduce the influence of a data distribution system between the clients under one-to-many conditions.
After step 102 is performed, an alternative embodiment is as follows: if the server side still does not receive the response message when the sending time is longer than or equal to the response waiting time, the server side determines that the target client side fails and stores the data to be transmitted in the cache space into a persistent storage space; and after the server determines that the target client has fault repaired, sending the data to be transmitted in the persistent storage space to the target client.
If the sending duration exceeds 10 seconds and the response message is still not received, the server determines that the target client fails. And the data to be transmitted in the cache space is still stored in a persistent storage space, and the data to be transmitted in the persistent storage space is sent to the target client after the server determines that the target client is subjected to fault repair.
The transmission mode may be set in the methods of steps 101 to 103. In an alternative embodiment (hereinafter referred to as an alternative embodiment of the transmission mode), the server maintains the transmission mode to the target client as the normal mode in a stage in which the failure of the target client is not determined; if the server determines that the target client fails, the sending mode is switched to a floor distribution mode; the floor distribution mode refers to that the server side sends data in the persistent storage space to the target client side.
That is, in the normal mode, the server sends the data in the cache space to the target client, and in the floor distribution mode, the data in the persistent storage space is sent to the target client.
Specifically, the method can realize the effect of automatic isolation and optimization, when one target client side is abnormal, the server side can automatically switch to a floor distribution mode, data in the cache space is written into the persistent storage space, and meanwhile, a trigger monitoring alarm mechanism can be used for reminding that the target client side has faults to wait for repair.
Based on the alternative embodiments of the transmission mode, one alternative embodiment is as follows:
and if the server determines that all the data in the persistent storage space are successfully transmitted to the corresponding client, switching the transmission mode back to the normal mode.
Specifically, after the target client is restored, the server reads data from the persistent storage space and sends the data to the client, and the server automatically switches back to the normal mode until the data in the persistent storage space is completely distributed.
An alternative embodiment of step 103 is as follows:
the server stores the data to be transmitted into a data file, and generates a file identification of the data file according to the timestamp of the data to be transmitted and the data abstract of the data to be transmitted; and the server stores the data file and the mapping relation between the data file and the file identifier into the persistent storage space.
The data file generates a file identifier of the data file according to the timestamp of the data to be transmitted and the data abstract of the data to be transmitted, and stores the mapping relation between the data file and the file identifier, so that the data file has traceability and can be used for developing other functions.
An alternative embodiment after step 103 is performed is as follows:
and if the server side determines that the plurality of clients successfully receive the data to be transmitted, deleting the data to be transmitted from the persistent storage space.
It should be noted that, in the above optional embodiments, the plurality of clients need to successfully receive the data to be transmitted, so that it is ensured that the data to be transmitted does not need to be generated any more, and the persistent storage space is saved.
An alternative embodiment of steps 101 to 103 is as follows:
if the server receives the response message in the sending time length is smaller than the response waiting time length, determining whether the response message indicates the target client to successfully respond or not; if the successful response is indicated, the server deletes the data to be transmitted from the cache space; and if the response is not successful, the server resends the data to be transmitted.
Specifically, the normal sending module in the normal mode, the floor distributing module in the floor distributing mode and the floor distributing response module can be set in the server, and the floor distributing module is called when the server sends data. After the floor distribution module receives the call of the server, the floor distribution module is called again to retry through the data distribution system from the persistent storage space, and the process of the retry can be realized through a universal timeout mechanism.
And the floor distribution response module receives the response message returned by the target client, marks that the data to be transmitted is successfully sent to the target task if the response message is successful, removes the data to be transmitted in the cache space, and can remove the data to be transmitted to a backup catalog for subsequent verification. If the response is not successful, the response is directly reported by mistake and discarded, and the server-side universal timeout mechanism is waited to recall the normal sending module and send again. If the successful response is indicated, the server deletes the data to be transmitted from the cache space; and if the response is not successful, the server resends the data to be transmitted.
Further, the overall design flow of the present solution may be as shown in fig. 2, in combination with the method shown in fig. 1 and the optional method.
a) In the normal mode, the server receives the data sent by the data source, stores the data in a designated cache block in a cache space (for example, a designated partition in Kafka), judges whether the state of the target client is normal, if the state is normal, normally distributes the data, and ends the exit; otherwise, continuing the process b.
b) And switching the server distribution mode to a floor distribution mode, and writing the data transmitted by the data source end which is received later into the persistent storage space to continue the c processing.
c) Updating the file identification of the data file of the data to be transmitted in the persistent storage space, then calling the floor distribution module to transmit the data file, judging whether the calling of the transmission module is overtime, and if the calling of the transmission module is overtime, calling the transmission module again to retransmit; otherwise, continuing the process d.
d) Calling a floor distribution response module, judging whether the response module is successfully called, and if the response module is failed, performing c processing; otherwise, enter e processing.
e) And judging whether the unsent data exists in the persistent storage space, and if so, executing the c process on the unsent data. If not, judging that the data is completely complemented when the consumer fails, recovering the state of the server to be in a normal distribution mode, and ending the exit.
As shown in fig. 3, the present application provides a data transmission apparatus, including: the acquisition module 301 is configured to acquire data to be transmitted from a data source end, and store the data to be transmitted into a buffer space; a processing module 302, configured to send the data to be transmitted to the target client if it is determined that the target client has not failed; in the stage that the sending duration is less than the response waiting duration and the response message of the target client is not received, keeping the data to be transmitted stored in the cache space; the target client is any one of a plurality of clients which are to consume the data to be transmitted; or if the target client fails, storing the data to be transmitted in the cache space into a persistent storage space; and after the fault repair of the target client is determined, sending the data to be transmitted in the persistent storage space to the target client.
Optionally, the processing module 302 is further configured to: if the response message is not received when the sending time length is longer than or equal to the response waiting time length, determining that the target client fails, and storing the data to be transmitted in the cache space into a persistent storage space; and after the fault repair of the target client is determined, sending the data to be transmitted in the persistent storage space to the target client.
Optionally, the processing module 302 is further configured to: maintaining a normal mode of a transmission mode to the target client in a stage in which the target client is not determined to be faulty; the processing module 302 is specifically configured to: if the target client side is determined to have a fault, switching the sending mode to a floor distribution mode; the floor distribution mode refers to sending data in the persistent storage space to the target client.
Optionally, the processing module 302 is further configured to: and if the data in the persistent storage space are determined to be successfully transmitted to the corresponding client, switching the transmission mode back to the normal mode.
Optionally, the processing module 302 is further configured to: and if the plurality of clients are determined to successfully receive the data to be transmitted, deleting the data to be transmitted from the persistent storage space.
Optionally, the processing module 302 is specifically configured to: storing the data to be transmitted into a data file, and generating a file identification of the data file according to the timestamp of the data to be transmitted and the data abstract of the data to be transmitted; and storing the data file and the mapping relation between the data file and the file identifier into the persistent storage space.
Optionally, the processing module 302 is further configured to: if the sending duration is smaller than the response waiting duration, the response message is received, and whether the response message indicates the target client to successfully respond or not is determined; if the successful response is indicated, deleting the data to be transmitted from the cache space; and if the response is not successful, retransmitting the data to be transmitted.
Based on the same inventive concept, the embodiments of the present application also provide a computer device, including a program or an instruction, which when executed by the computer device, performs the data processing method and any optional method as shown in fig. 1.
Based on the same inventive concept, the embodiments of the present application also provide a computer-readable storage medium including a program or instructions, which when run on a computer, cause the computer to perform the data processing method and any optional method shown in fig. 1.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, or as a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (9)
1. A data transmission method, comprising:
the method comprises the steps that a server side obtains data to be transmitted from a data source side and stores the data to be transmitted into a cache space;
if the server determines that the target client does not fail, the server sends the data to be transmitted to the target client; the server keeps the data to be transmitted stored in the cache space in a stage that the sending duration is smaller than the response waiting duration and the response message of the target client is not received; the target client is any one of a plurality of clients which are to consume the data to be transmitted; the server side keeps the sending mode of the target client side as a normal mode in a stage of not determining that the target client side fails; the normal mode refers to that the server side sends data in the cache space to the target client side; or,
if the server determines that the target client fails, storing the data to be transmitted in the cache space into a persistent storage space, and switching a sending mode of the target client into a floor distribution mode; the floor distribution mode refers to that the server side sends data in the persistent storage space to the target client side; and after the server determines that the target client has fault repaired, sending the data to be transmitted in the persistent storage space to the target client.
2. The method of claim 1, wherein the server, if determining that the target client has not failed, further comprises, after sending the data to be transmitted to the target client:
if the server side still does not receive the response message when the sending time is longer than or equal to the response waiting time, the server side determines that the target client side fails and stores the data to be transmitted in the cache space into a persistent storage space; and after the server determines that the target client has fault repaired, sending the data to be transmitted in the persistent storage space to the target client.
3. The method of claim 1, wherein after the server side determines that the target client fails, the server side switches the sending mode to the floor distribution mode, further comprising:
and if the server determines that all the data in the persistent storage space are successfully transmitted to the corresponding client, switching the transmission mode back to the normal mode.
4. A method according to any one of claims 1 to 3, wherein after said sending said data to be transmitted in said persistent storage space to said target client, further comprising:
and if the server side determines that the plurality of clients successfully receive the data to be transmitted, deleting the data to be transmitted from the persistent storage space.
5. A method according to any one of claims 1 to 3, wherein said storing said data to be transmitted in said cache space in said persistent storage space comprises:
the server stores the data to be transmitted into a data file, and generates a file identification of the data file according to the timestamp of the data to be transmitted and the data abstract of the data to be transmitted;
and the server stores the data file and the mapping relation between the data file and the file identifier into the persistent storage space.
6. A method as claimed in any one of claims 1 to 3, further comprising:
if the server receives the response message in the sending time length is smaller than the response waiting time length, determining whether the response message indicates the target client to successfully respond or not;
if the successful response is indicated, the server deletes the data to be transmitted from the cache space; and if the response is not successful, the server resends the data to be transmitted.
7. A data transmission apparatus, comprising:
the acquisition module is used for acquiring data to be transmitted from a data source end and storing the data to be transmitted into a cache space;
the processing module is used for sending the data to be transmitted to the target client if the target client is determined to not have a fault; in the stage that the sending duration is less than the response waiting duration and the response message of the target client is not received, keeping the data to be transmitted stored in the cache space; the target client is any one of a plurality of clients which are to consume the data to be transmitted; the method comprises the steps that a sending mode of a target client is kept to be a normal mode in a stage that the target client is not determined to be faulty; the normal mode refers to sending data in the cache space to the target client; or,
if the target client side is determined to be faulty, storing the data to be transmitted in the cache space into a persistent storage space, and switching a sending mode of the target client side into a floor distribution mode; the floor distribution mode refers to sending data in the persistent storage space to the target client; and after the fault repair of the target client is determined, sending the data to be transmitted in the persistent storage space to the target client.
8. A computer device comprising a program or instructions which, when executed, performs the method of any of claims 1 to 6.
9. A computer readable storage medium comprising a program or instructions which, when executed, performs the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010697261.9A CN111880947B (en) | 2020-07-20 | 2020-07-20 | Data transmission method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010697261.9A CN111880947B (en) | 2020-07-20 | 2020-07-20 | Data transmission method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111880947A CN111880947A (en) | 2020-11-03 |
| CN111880947B true CN111880947B (en) | 2023-12-15 |
Family
ID=73155845
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010697261.9A Active CN111880947B (en) | 2020-07-20 | 2020-07-20 | Data transmission method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111880947B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113034283A (en) * | 2021-03-29 | 2021-06-25 | 中国银联股份有限公司 | Transaction data processing method, device and system |
| CN113783939B (en) * | 2021-08-20 | 2024-09-03 | 奇安信科技集团股份有限公司 | File transmission method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105933380A (en) * | 2016-04-01 | 2016-09-07 | 宇龙计算机通信科技(深圳)有限公司 | Data transmission method, device and terminal |
| CN110505275A (en) * | 2019-07-16 | 2019-11-26 | 北京奇艺世纪科技有限公司 | Data transmission method, system, server and computer readable storage medium |
| CN110795166A (en) * | 2019-10-28 | 2020-02-14 | 深圳前海微众银行股份有限公司 | Data processing method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10425483B2 (en) * | 2016-05-16 | 2019-09-24 | International Business Machines Corporation | Distributed client based cache for keys using demand fault invalidation |
| US10802972B2 (en) * | 2018-08-02 | 2020-10-13 | MemVerge, Inc | Distributed memory object apparatus and method enabling memory-speed data access for memory and storage semantics |
-
2020
- 2020-07-20 CN CN202010697261.9A patent/CN111880947B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105933380A (en) * | 2016-04-01 | 2016-09-07 | 宇龙计算机通信科技(深圳)有限公司 | Data transmission method, device and terminal |
| CN110505275A (en) * | 2019-07-16 | 2019-11-26 | 北京奇艺世纪科技有限公司 | Data transmission method, system, server and computer readable storage medium |
| CN110795166A (en) * | 2019-10-28 | 2020-02-14 | 深圳前海微众银行股份有限公司 | Data processing method and device |
Non-Patent Citations (1)
| Title |
|---|
| 面向QEMU的分布式块存储系统的设计与实现;张沪滨等;《微型电脑应用》;第32卷(第04期);第54-57页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111880947A (en) | 2020-11-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105933407B (en) | method and system for realizing high availability of Redis cluster | |
| US9325757B2 (en) | Methods and systems for fault-tolerant distributed stream processing | |
| US9189348B2 (en) | High availability database management system and database management method using same | |
| US20130205017A1 (en) | Computer failure monitoring method and device | |
| CN111880947B (en) | Data transmission method and device | |
| JP5343436B2 (en) | Information management system | |
| CN112433885A (en) | Block chain consensus processing method and device, electronic equipment and storage medium | |
| CN114422335B (en) | Communication method, device, server and storage medium | |
| CN117640747B (en) | Vhost-user reconnection method and system based on proxy service | |
| CN112948484A (en) | Distributed database system and data disaster recovery drilling method | |
| CN113467953B (en) | Service state switching method and device, server and storage medium | |
| CN117762652A (en) | Distributed transaction processing method and device based on message middleware | |
| CN107943567B (en) | High-reliability task scheduling method and system based on AMQP protocol | |
| JP2005301436A (en) | Cluster system and failure recovery method for it | |
| CN116185697B (en) | Container cluster management method, device and system, electronic equipment and storage medium | |
| CN115599310B (en) | Method and device for controlling storage resources in storage node and storage node | |
| CN110351122B (en) | Disaster recovery method, device, system and electronic equipment | |
| CN112131185B (en) | Method and device for high availability of service in super-fusion distributed storage node | |
| KR100832890B1 (en) | Process obstacle lookout method and recovery method for information communication | |
| CN108599903B (en) | Single board starting control method and device | |
| CN104796228B (en) | A kind of method, apparatus and system of information transmission | |
| CN113032106A (en) | Automatic detection method and device for IO suspension abnormality of computing node | |
| US20210247996A1 (en) | Service continuation system and service continuation method | |
| CN115190005B (en) | Redis-based high availability method of double-host system | |
| CN110134628B (en) | Message transmission method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |