CN113965482A - Data transmission method and device based on gRPC and storage medium - Google Patents
Data transmission method and device based on gRPC and storage medium Download PDFInfo
- Publication number
- CN113965482A CN113965482A CN202111216434.1A CN202111216434A CN113965482A CN 113965482 A CN113965482 A CN 113965482A CN 202111216434 A CN202111216434 A CN 202111216434A CN 113965482 A CN113965482 A CN 113965482A
- Authority
- CN
- China
- Prior art keywords
- data
- transmitted
- client
- data packet
- transmission
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0823—Errors, e.g. transmission errors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0823—Errors, e.g. transmission errors
- H04L43/0829—Packet loss
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0852—Delays
- H04L43/087—Jitter
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Communication Control (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application relates to a data transmission method, a device and a storage medium based on gRPC, belonging to the field of data communication, wherein the method comprises that a client and a server judge whether an information channel is available or not based on a preset verification rule under the gRPC environment; when an information channel between the client and the server is unavailable, caching data to be transmitted; when an information channel between the client and the server is available, transmitting the data to be transmitted, and judging whether the data to be transmitted is transmitted successfully; if the transmission of the data to be transmitted fails, performing transmission retry; and when the transmission retry times are larger than a preset retry time threshold value, caching the data to be transmitted. When the information channel is unavailable, all data to be transmitted are cached, transmission is facilitated when the information channel is available, the data to be transmitted are not easy to lose, and the data transmission quality is improved.
Description
Technical Field
The invention belongs to the field of data communication, and particularly relates to a data transmission method and device based on a gPC (gateway GPRS), and a storage medium.
Background
The gRPC is a language neutral, platform neutral, and open source remote procedure call system. At present, probes in a website monitoring system are mostly deployed in an extranet, and the extranet has a complex network environment and certain instability. When the extranet probe transmits data to the website monitoring cloud server through the gPC, if unstable conditions such as network jitter occur, the phenomena of blocking, overtime, packet loss and the like of data transmission can be caused, and the quality of data transmission is influenced.
Disclosure of Invention
The invention provides a data transmission method, a data transmission device and a storage medium based on a gPRC (generalized guide pc) in order to improve the data transmission quality.
In a first aspect, the data transmission method based on the gRPC provided by the present application adopts the following technical scheme:
a data transmission method based on gRPC includes:
the client and the server judge whether the information channel is available or not based on a preset verification rule under the gPC environment;
when an information channel between the client and the server is unavailable, caching data to be transmitted;
when an information channel between the client and the server is available, transmitting the data to be transmitted, and judging whether the data to be transmitted is transmitted successfully;
if the transmission of the data to be transmitted fails, performing transmission retry;
and when the transmission retry times are larger than a preset retry time threshold value, caching the data to be transmitted.
By adopting the technical scheme, when the information channel is unavailable, all data to be transmitted are cached, the data can be transmitted conveniently when the information channel is available, the data to be transmitted is not easy to lose, and the data transmission quality is improved. When the information channel is available, whether the data to be transmitted is successfully transmitted or not is monitored in real time, if the data to be transmitted is unsuccessful, the fact that the network fluctuates or the transmission environment changes is proved, at the moment, transmission retry is conducted on the data to be transmitted, the loss rate of the data to be transmitted is reduced, and the data transmission quality is further improved.
Optionally, the step of determining, by the client and the server, whether the information channel is available based on a preset validation rule in the gRPC environment includes:
the client sends a preset first data packet to the server, updates the client state identifier to be ready, and stores the updated client state identifier;
the server receives the first data packet, returns the received first data packet and a preset second data packet to the client together, and stores the updated client state identifier;
the client receives the first data packet and the second data packet and judges whether the stored client state identifier is ready;
if so, updating the client state identifier again to enable the client state identifier to be connected, and returning the received first data packet, the second data packet and a preset third data packet to the server;
the server receives the first data packet, the second data packet and the third data packet, and judges whether the stored client state identifier is ready;
if so, changing the stored client state identifier into a stable state, and returning the received first data packet, the second data packet and the third data packet to the client;
the client receives the first data packet, the second data packet and the third data packet, and judges whether the stored client state identifier is connected or not;
if yes, updating the client state identifier three times to enable the client state identifier to be stable, and meanwhile judging that the information channel is available.
By adopting the technical scheme, the preset first data packet, the preset second data packet and the preset third data packet are mutually transmitted between the client and the server, on one hand, the preset data packet has a certain size and is similar to data to be transmitted, whether the information channel is available or not is verified in a way of mutually transmitting the data packets, and the accuracy is higher; on the other hand, a new data packet is added in each transmission, the client state identification is updated after the data packet is received, whether the information channel is really available or not is judged conveniently based on the size or the number of the received data packets and the client state identification, whether the data packet and the client state identification are abnormal or not in the transmission process is judged conveniently, the data packet and the client state identification are not abnormal in two aspects, and the data packet and the client state identification are regarded as being available in the transmission process, so that the data transmission quality is guaranteed.
Optionally, after the transmitting the data to be transmitted, the method further includes:
the client transmits the first data packet, the second data packet and the third data packet to the server, and judges whether the transmission exceeds a preset maintenance time threshold or whether to receive the information which is transmitted by the server and refuses to receive;
if the transmission exceeds a preset maintenance time threshold or the receiving refusing information transmitted by the server is received, judging that the information channel is abnormal, and judging whether the information channel is available again;
and if the times of the re-judgment exceed the threshold value of the reconstruction times, the information channel is regarded as unavailable.
By adopting the technical scheme, after the information channel is judged to be available, whether the information channel is available or not is verified in real time, and on one hand, when the data to be transmitted cannot be transmitted or is abnormal in transmission, whether the problem of the information channel or the problem of the data to be transmitted is caused is easy to know, so that the data to be transmitted is not easy to lose, and the repair efficiency is improved; and then, the unavailability of the information channel is easy to find in time, so that the transmission action of the data to be transmitted is stopped, the data to be transmitted is cached, the loss rate of the data to be transmitted is favorably reduced, and the data transmission quality is improved.
Optionally, the step of determining whether the data to be transmitted is successfully transmitted includes:
the client judges whether the transmission success information fed back by the server exceeds a preset feedback time threshold value;
if yes, the transmission is regarded as failure;
otherwise, the transmission is considered to be successful.
By adopting the technical scheme, after the client transmits the data to be transmitted to the server, the server feeds back corresponding transmission success information to the client, and the client judges whether the data to be transmitted is successfully transmitted or not based on the time of the received transmission success information, so that the client can conveniently make a judgment in time and prevent a large amount of data to be transmitted from being transmitted to the server under an abnormal condition. In addition, whether the data to be transmitted is successfully transmitted or not can be judged in time, and the phenomena of blocking and overtime of the data to be transmitted are reduced.
Optionally, if the transmission of the data to be transmitted fails, the step of performing transmission retry includes:
retransmitting the current data to be transmitted, and stopping transmitting the subsequent data to be transmitted;
the number of retries is increased by 1.
By adopting the technical scheme, when the data to be transmitted is retransmitted, the subsequent data to be transmitted is not transmitted, so that the loss rate of the data to be transmitted is reduced, and the data transmission quality is improved.
Optionally, the step of caching the data to be transmitted includes:
storing the data to be transmitted to a preset message queue;
and after receiving the processing information, taking out the corresponding data to be transmitted from the message queue, and transmitting the corresponding data to be transmitted.
By adopting the technical scheme, the data to be transmitted is stored in the message queue, so that the data to be transmitted can be stored in order and is not easy to lose. After the processing information is received, the data to be transmitted is taken out from the message queue, so that the data to be transmitted can be output orderly and is not easy to lose.
Optionally, the message queue includes a first queue and a second queue;
the step of storing the data to be transmitted to a preset message queue comprises:
storing the data to be transmitted to the first queue;
after receiving the processing information, the step of taking out the corresponding data to be transmitted from the message queue and transmitting the corresponding data to be transmitted comprises the following steps:
transmitting the data to be transmitted stored in the first queue based on a first-in first-out rule;
judging whether the data to be transmitted is transmitted successfully;
if the data to be transmitted in the first queue is successful, traversing the data to be transmitted in the first queue;
if the data to be transmitted fails to be transmitted, storing the data to be transmitted which fails to be transmitted to the second queue, and traversing the data to be transmitted in the first queue;
and moving the data to be transmitted in the second queue into the first queue.
By adopting the technical scheme, after the data to be transmitted in the first queue is successfully transmitted, all the data to be transmitted in the first queue are transmitted once, and the data which is failed to be transmitted is stored in the second queue. After the data to be transmitted in the first queue are transmitted, the data to be transmitted in the second queue are transferred into the first queue to wait for the next processing information, so that the transmission efficiency of the data to be transmitted is improved.
In a second aspect, the present application provides a gRPC-based data transmission apparatus, which employs the following technical solutions:
a data transmission device based on a gPC comprises a memory and a processor, wherein the memory stores a data transmission program based on the gPC; the processor is configured to employ the above-described method in executing a gRPC-based data transmission program.
By adopting the technical scheme, the memory stores the data transmission program based on the gRPC, and the processor can execute the program by adopting the method, thereby being beneficial to improving the data transmission quality.
In a third aspect, the present application provides a storage medium, which adopts the following technical solutions:
a storage medium stores a computer program that can be loaded by a processor and that executes the above-described method.
By adopting the technical scheme, the data transmission program is stored, which is beneficial to improving the data transmission quality.
In summary, the information channel is preferentially judged, when the information channel is available, the data to be transmitted is transmitted, and when the information channel is unavailable, the data to be transmitted is cached, so that the data to be transmitted is not easy to lose, and the data transmission quality is favorably improved.
In addition, when the data to be transmitted is transmitted, whether the data to be transmitted is transmitted successfully or not is judged, so that the accurate circulation of the data to be transmitted from the output end to the receiving end is ensured, the data to be transmitted is not easy to steal or tamper midway, and the data transmission quality is improved.
And if the transmission retry times are greater than a preset retry time threshold value, caching the data to be transmitted to help ensure that the data to be transmitted is always in a stable transmission environment, otherwise, not transmitting the data, and helping to improve the transmission quality of the data.
Drawings
Fig. 1 is a flowchart of a data transmission method based on a gRPC according to an embodiment of the present application.
Fig. 2 is a flowchart of determining whether an information channel is available according to an embodiment of the present application.
Fig. 3 is another flowchart of determining whether an information channel is available according to an embodiment of the present application.
Fig. 4 is a flowchart illustrating a step 200 of a gRPC-based data transmission method according to an embodiment of the present disclosure.
Detailed Description
The embodiment of the application discloses a data transmission method based on gRPC. Referring to fig. 1, includes:
s100, the client and the server judge whether the information channel is available or not based on a preset verification rule under the gPC environment.
In the gRPC environment, the client and the server perform data transmission by using the gRPC technology, that is, the client and the server communicate in a remote procedure call system such as the gRPC.
Specifically, in an embodiment, referring to fig. 2, step S100 includes:
s111, the client sends a preset first data packet to the server, meanwhile, the client state identifier is updated to be ready, and the updated client state identifier is stored.
For ease of understanding, the client state identifier may be a null value when the client does not send the first packet, and the client state identifier becomes ready when the client sends the first packet or after the client sends the first packet. The updated client status flag is stored in step S111, i.e. it is ready to be stored.
And S112, the server receives the first data packet, returns the received first data packet and a preset second data packet to the client side, and stores the updated client state identifier.
When the method is applied, the client and the server can be pre-stored with the first data packet and the second data packet, and calling can be carried out according to the situation. Of course, if the data to be transmitted can only be sent to the server side by the client side, only the first data packet can be prestored in the client side, and only the second data packet can be prestored in the server side, so that network resources are saved.
The server returns the received first data packet and the preset second data packet to the client together, namely the server fuses or binds the first data packet and the second data packet and transmits the fused or bound first data packet and second data packet to the client. For convenience of understanding, for example, if the first data packet is a and the second data packet is b, the server transmits a + b to the client after receiving a. The client state identifier stored by the server may be the client state identifier which is updated when the client transmits the first data packet to the server; the client side can also store the updated client state identifier into a first data packet, and the server reads the information in the first data packet to obtain the updated client state identifier; or when the server receives the first data packet, the server automatically judges the client state identifier as ready based on the preset corresponding relation, and then stores the client state identifier.
S113, the client receives the first data packet and the second data packet and judges whether the stored client state identifier is ready.
If not, judging that the information channel is unavailable;
and if so, updating the client state identifier again to enable the client state identifier to be connected, and returning the received first data packet, the second data packet and the preset third data packet to the server.
The third data packet may be stored only in the server, or both the client and the server may store the third data packet in advance, as in the case of the first data packet and the second data packet.
S114, the server receives the first data packet, the second data packet and the third data packet, and judges whether the stored client state identification is ready.
If not, judging that the information channel is unavailable;
if yes, the stored client state representation is changed to be stable, and the received first data packet, the received second data packet and the received third data packet are returned to the client.
S115, the client receives the first data packet, the second data packet and the third data packet, and judges whether the stored client state identifier is connected.
If not, judging that the information channel is unavailable;
and if so, updating the client state identifier for the third time to ensure that the client state identifier is stable and the information channel is judged to be available.
As for steps S112 to S115, it should be noted that, in addition to the client status flag being checked, if the client or the server receives an error data packet or an error data packet quantity in the corresponding step, an abnormal alarm message is output or an information channel is directly determined to be unavailable. For example, in the execution process of step S112, the server determines whether the received first data packet is the same as a locally pre-stored first data packet, where the same refers to the same content and data structure; if the data packets are the same, judging whether the received data packets only have the first data packets, if so, executing subsequent steps, and otherwise, outputting abnormal alarm information or judging that the information channel is unavailable. For another example, in the process of not being executed by S113, the client determines whether the received first data packet and second data packet are the same as the first data packet and second data packet pre-stored locally; if the first data packet, the second data packet and the third data packet are received, outputting abnormal alarm information or judging that an information channel is unavailable, otherwise, outputting abnormal alarm information or judging that the information channel is unavailable.
In another embodiment, referring to fig. 3, the data to be transmitted may be initiated by the server to the client, where step S100 includes:
s121, the server sends a preset first data packet to the client, meanwhile, the service state is updated to be ready, and the updated client state identifier is stored.
And S122, the client receives the first data packet, returns the received first data packet and a preset second data packet to the server together, and stores the updated service state identifier.
S123, the server receives the first data packet and the second data packet and judges whether the stored service state identifier is ready.
If not, judging that the information channel is unavailable;
and if so, updating the user state identifier again to enable the user state identifier to be connected, and returning the received first data packet, the received second data packet and the received third data packet to the server.
S124, the client receives the first data packet, the second data packet and the third data packet, and judges whether the stored service state identifier is ready.
If not, judging that the information channel is unavailable;
if yes, the stored service state representation is changed to be stable, and the received first data packet, the received second data packet and the received third data packet are returned to the server.
S125, the server receives the first data packet, the second data packet and the third data packet, and judges whether the stored service state identifier is connected.
If not, judging that the information channel is unavailable;
and if so, updating the service state identifier for the third time to ensure that the service state identifier is stable, and meanwhile, judging that the information channel is available.
The principle of steps S121-S125 is the same as that of steps S111-S115, and is not described in detail.
Referring to fig. 1, when the information channel between the client and the server is unavailable, step S200 is performed to cache the data to be transmitted.
When the information channel between the client and the server is available, step S300 is executed to transmit the data to be transmitted, and determine whether the data to be transmitted is successfully transmitted.
In an embodiment, the condition that the data to be transmitted is initiated by the client and transmitted to the server further includes, after the data to be transmitted is transmitted:
and the client transmits the first data packet, the second data packet and the third data packet to the server, and judges whether the transmission exceeds a preset maintenance time threshold or not, or judges whether the receiving refusing information transmitted by the server is received or not.
When the transmission time exceeds the maintenance time threshold value, namely the client transmits the first data packet, the second data packet and the third data packet to the server, timing is started; and when the successful feedback information returned by the server is received, stopping timing. And judging whether the timing time exceeds a maintaining time threshold value.
If the client judges that the transmission exceeds a preset maintaining time threshold or receives the information refusing to receive transmitted by the server, the client judges that the information channel is abnormal and judges whether the information channel is available again.
If the timing time exceeds the maintaining time threshold value after the timing is started, but any information transmitted by the server is not received, the information channel is directly judged to be abnormal. The steps S111-S115 are performed to re-determine whether the information channel is available.
If the number of times of re-determining whether the information channel is available exceeds the threshold of the number of times of reconstruction, the information channel is considered to be unavailable, the transmission of the data to be transmitted is stopped, and step S200 is executed.
For example, after determining that the information channel is abnormal, the client performs steps S111 to S115 again with the server, but the result is that the information channel is not available. In order to ensure the transmission timeliness of the data to be transmitted, the client and the server will perform steps S111-S115 again, and repeatedly confirm whether the information channel is available. Adding 1 to the number of times of judging whether the information channel is available or not in each repeated confirmation, and when the number of times exceeds a reconstruction number threshold, judging the availability of the information channel no longer; for example, the threshold number of reconstructions is 3.
In another embodiment, in the case that the data to be transmitted is initiated by the server and transmitted to the client, after the data to be transmitted is transmitted, the method further includes:
and the server transmits the first data packet, the second data packet and the third data packet to the client, and judges whether the transmission exceeds a preset maintenance time threshold or not, or judges whether receiving refusal receiving information transmitted by the client is received or not.
If the server judges that the transmission exceeds a preset maintaining time threshold or receives the information refusing to receive transmitted by the server, the server judges that the information channel is abnormal and judges whether the information channel is available again.
Steps S121-S125 are performed to determine whether the information channel is available again.
If the number of times of re-determining whether the information channel is available exceeds the threshold of the number of times of reconstruction, the information channel is considered to be unavailable, the transmission of the data to be transmitted is stopped, and step S200 is executed.
It should be noted that, no matter the data to be transmitted is transmitted from the client to the server or from the server to the client, as long as the data to be transmitted is in the transmission state and the transmission failure of the data to be transmitted is not determined, whether the information channel is abnormal or not is continuously determined.
The step of judging whether the data to be transmitted is transmitted successfully comprises the following steps:
the client judges whether the transmission success information fed back by the server exceeds a preset feedback time threshold. If yes, the transmission is regarded as failure; if not, the transmission is considered to be successful.
It is understood that, when the client transmits one piece of data to be transmitted to the server, the server returns a piece of corresponding transmission success information to the client. When the client sends a piece of data to be transmitted to the server, timing is started for the piece of data to be transmitted, and the timing is stopped after the transmission success information returned by the server for the piece of data to be transmitted is received; and comparing the timing time with the feedback time threshold value, and judging whether the data to be transmitted is transmitted successfully. To help reduce the loss rate of data to be transmitted, the feedback time threshold is less than the total time it takes for the client to send two or three pieces of data to be transmitted to the server. For example, if the client transmits one piece of data to be transmitted to the server every 10ms, the feedback time threshold may be set to 15ms, 20ms, or 25 ms.
For the case that the data to be transmitted is transmitted from the server to the client, if the principle and the specific implementation manner of judging whether the data to be transmitted is successfully transmitted are the same as those in the above case, the details are not repeated.
Referring to fig. 1, if the data to be transmitted is successfully transmitted, the data to be transmitted is continuously transmitted; if the transmission of the data to be transmitted fails, step S400 is executed to perform a transmission retry.
Specifically, step S400 specifically includes:
and retransmitting the current data to be transmitted, stopping transmitting subsequent data to be transmitted, recording the retry times, and increasing the retry times by 1 before retransmitting each time.
The current data to be transmitted refers to the data to be transmitted which fails to be transmitted, and the subsequent data to be transmitted refers to the data to be transmitted which is not transmitted yet. Once the retransmission is made, the number of retries is increased by 1.
S500, judging whether the transmission retry number is larger than a preset retry number threshold value.
When the number of transmission retries is greater than the preset threshold of the number of retries, step S200 is executed to cache the data with transmission.
Specifically, referring to fig. 4, step S200 includes:
s210, storing the data to be transmitted to a preset message queue.
Wherein the message queue comprises a first queue and a second queue, and the first queue and the second queue can be RabbitMQ, RocketMQ, Kafka or Redis.
S220, after receiving the processing information, taking out the corresponding data to be transmitted from the message queue, and transmitting the corresponding data to be transmitted.
Specifically, the processing information may be generated or received by the client or the server after determining that the information channel is available, or after the data to be transmitted can be successfully transmitted. The execution main body for taking out the corresponding data to be transmitted from the message queue is determined by the output party of the data to be transmitted; that is, if the data to be transmitted is transmitted from the client to the server, the client performs the fetching action, otherwise, the server performs the fetching action.
The specific steps of storing the data to be transmitted to a preset message queue include:
and storing the data to be transmitted to the first queue.
After receiving the processing information, the specific steps of taking out the corresponding data to be transmitted from the message queue and transmitting the corresponding data to be transmitted include:
the data to be transmitted stored in the first queue is transmitted based on a first-in-first-out rule.
And judging whether the data to be transmitted is transmitted successfully or not.
The determination method is the same as that described above, and is not described in detail.
And if the data is successful, traversing the data to be transmitted in the first queue.
It should be noted that, the data to be transmitted is read from the first queue based on the first-in first-out rule. The data to be transmitted is transmitted piece by piece. At this time, whether the transmission of the data to be transmitted fails or not, the data to be transmitted in the first queue is transmitted once according to the first-in first-out order.
If the data transmission is unsuccessful, the data to be transmitted which fails to be transmitted is stored in the second queue, and the data to be transmitted in the first queue is traversed.
For example, if the transmission of the first piece of data to be transmitted in the first queue fails, the first piece of data to be transmitted is imported from the first queue to the second queue, and the second piece of data to be transmitted is transmitted, and so on until all the data to be transmitted in the first queue is transmitted.
And moving the data to be transmitted in the second queue into the first queue.
For further understanding of the present solution, the following describes an embodiment of a data transmission method in conjunction with a specific implementation scenario.
The website monitoring platform monitors a target website based on a probe deployed on an external network. The specific monitoring process is as follows: the probe acquires a monitoring task from a website monitoring platform, wherein the monitoring task comprises a website to be monitored, namely a target website; the monitoring period, namely the time for starting monitoring and the frequency for monitoring; and the monitoring strategy is to monitor the contents of the target website and the layer number and the page number of the target website. After the probe obtains the monitoring task, information in the target website is captured, and captured data are returned to the website monitoring platform; and the website monitoring platform monitors the target website based on the data returned by the probe.
In this embodiment, the website monitoring platform is equivalent to a server, and the probes are equivalent to clients, so it is understood that the website monitoring platform generally needs to monitor a plurality of target websites at the same time, and therefore the number of the probes communicating with the website monitoring platform is multiple.
When the target website is monitored, the website monitoring platform establishes a monitoring task locally and waits for a probe to be picked up.
The probe firstly judges whether the information channel is available or not based on a preset verification rule. The probe end and the website monitoring platform end are pre-stored with a first data packet, a second data packet and a third data packet, and the probe transmits the pre-stored data packets with the website monitoring platform according to the verification rule, so that whether an information channel between the probe end and the website monitoring platform is available is judged. The process can also be understood as the probe establishes an information channel with the website monitoring platform.
When the information channel is available, the probe packages the self information and transmits the self information to the website monitoring platform, and starts overtime waiting. The self information of the probe comprises a probe ID, a probe type, a pulling task type, a matching label and the like.
And after receiving the self information uploaded by the probe, the website monitoring platform returns the corresponding monitoring task data to the probe.
And if the probe receives monitoring task data returned by the website monitoring platform before the timeout, starting to execute the monitoring task.
In the data transmission process, if the information channel is unavailable, the probe caches the self information needing to be transmitted to the website monitoring platform; if the probe does not receive the monitoring task data returned by the website monitoring platform before the timeout, the data transmission is regarded as failed, and then transmission retry is carried out. If the transmission retry times are larger than the preset retry time threshold, caching the self information.
It is easy to understand that besides the information of the probe itself, when other data are transmitted, the determination of whether the information channel is available, the determination of whether the data transmission is successful and the determination of the transmission retry number can be performed, which is helpful to establish a stable information channel between the probe and the website monitoring platform, to eliminate the data transmission failure fault caused by network oscillation, and to reduce the data loss rate.
The embodiment also discloses a data transmission device based on the gPC, which comprises a memory and a processor, wherein the memory stores a data transmission program based on the gPC, and the processor is used for adopting the data transmission method when executing the data transmission program based on the gPC.
The embodiment also discloses a storage medium, which stores a computer program that can be loaded by a processor and executes the data transmission method.
The above embodiments are preferred embodiments of the present application, and the protection scope of the present application is not limited by the above embodiments, so: all equivalent changes made according to the structure, shape and principle of the present application shall be covered by the protection scope of the present application.
Claims (9)
1. A data transmission method based on gRPC is characterized by comprising the following steps:
the client and the server judge whether the information channel is available or not based on a preset verification rule under the gPC environment;
when an information channel between the client and the server is unavailable, caching data to be transmitted;
when an information channel between the client and the server is available, transmitting the data to be transmitted, and judging whether the data to be transmitted is transmitted successfully;
if the transmission of the data to be transmitted fails, performing transmission retry;
and when the transmission retry times are larger than a preset retry time threshold value, caching the data to be transmitted.
2. The data transmission method based on the gRPC of claim 1, wherein the step of the client and the server determining whether the information channel is available based on a preset validation rule in the gRPC environment includes:
the client sends a preset first data packet to the server, updates the client state identifier to be ready, and stores the updated client state identifier;
the server receives the first data packet, returns the received first data packet and a preset second data packet to the client together, and stores the updated client state identifier;
the client receives the first data packet and the second data packet and judges whether the stored client state identifier is ready;
if so, updating the client state identifier again to enable the client state identifier to be connected, and returning the received first data packet, the second data packet and a preset third data packet to the server;
the server receives the first data packet, the second data packet and the third data packet, and judges whether the stored client state identifier is ready;
if so, changing the stored client state identifier into a stable state, and returning the received first data packet, the second data packet and the third data packet to the client;
the client receives the first data packet, the second data packet and the third data packet, and judges whether the stored client state identifier is connected or not;
if yes, updating the client state identifier three times to enable the client state identifier to be stable, and meanwhile judging that the information channel is available.
3. A gRPC-based data transmission method according to claim 2, further comprising, after the transmitting the data to be transmitted:
the client transmits the first data packet, the second data packet and the third data packet to the server, and judges whether the transmission exceeds a preset maintenance time threshold or whether to receive the information which is transmitted by the server and refuses to receive;
if the transmission exceeds a preset maintenance time threshold or the receiving refusing information transmitted by the server is received, judging that the information channel is abnormal, and judging whether the information channel is available again;
and if the times of the re-judgment exceed the threshold value of the reconstruction times, the information channel is regarded as unavailable.
4. The gRPC-based data transmission method of claim 1, wherein the step of determining whether the data to be transmitted is successfully transmitted comprises:
the client judges whether the transmission success information fed back by the server exceeds a preset feedback time threshold value;
if yes, the transmission is regarded as failure;
otherwise, the transmission is considered to be successful.
5. A gRPC-based data transmission method as claimed in claim 1 or 4, wherein if the transmission of the data to be transmitted fails, the retry of transmission comprises:
retransmitting the current data to be transmitted, and stopping transmitting the subsequent data to be transmitted;
the number of retries is increased by 1.
6. A gRPC-based data transmission method according to claim 1, wherein the step of buffering data to be transmitted includes:
storing the data to be transmitted to a preset message queue;
and after receiving the processing information, taking out the corresponding data to be transmitted from the message queue, and transmitting the corresponding data to be transmitted.
7. A gRPC-based data transmission method as claimed in claim 6, wherein the message queue includes a first queue and a second queue;
the step of storing the data to be transmitted to a preset message queue comprises:
storing the data to be transmitted to the first queue;
after receiving the processing information, the step of taking out the corresponding data to be transmitted from the message queue and transmitting the corresponding data to be transmitted comprises the following steps:
transmitting the data to be transmitted stored in the first queue based on a first-in first-out rule;
judging whether the data to be transmitted is transmitted successfully;
if the data to be transmitted in the first queue is successful, traversing the data to be transmitted in the first queue;
if the data to be transmitted fails to be transmitted, storing the data to be transmitted which fails to be transmitted to the second queue, and traversing the data to be transmitted in the first queue;
and moving the data to be transmitted in the second queue into the first queue.
8. A data transmission device based on gRPC, its characterized in that: the gPRC-based data transmission device comprises a memory and a processor, wherein a gPRC-based data transmission program is stored in the memory; the processor is configured to employ the method of any one of claims 1-7 when executing a gRPC-based data transmission program.
9. A storage medium, characterized by: a computer program which can be loaded by a processor and which performs the method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111216434.1A CN113965482B (en) | 2021-10-19 | 2021-10-19 | Data transmission method and device based on gRPC and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111216434.1A CN113965482B (en) | 2021-10-19 | 2021-10-19 | Data transmission method and device based on gRPC and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113965482A true CN113965482A (en) | 2022-01-21 |
| CN113965482B CN113965482B (en) | 2023-03-24 |
Family
ID=79465368
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111216434.1A Active CN113965482B (en) | 2021-10-19 | 2021-10-19 | Data transmission method and device based on gRPC and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113965482B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114552314A (en) * | 2022-03-02 | 2022-05-27 | 深圳市爱普丰电子有限公司 | Data transmission method based on multi-head data line and data line thereof |
| CN114697376A (en) * | 2022-03-16 | 2022-07-01 | 浪潮云信息技术股份公司 | Super-large message transmission method and device |
| CN115334000A (en) * | 2022-07-20 | 2022-11-11 | 苏州浪潮智能科技有限公司 | Data transmission method, device, equipment and medium based on multi-control storage system |
| WO2023160276A1 (en) * | 2022-02-25 | 2023-08-31 | 腾讯科技（深圳）有限公司 | Data processing method and apparatus, computer device and readable storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1145496A2 (en) * | 1999-01-29 | 2001-10-17 | Sun Microsystems, Inc. | A method to monitor and control server applications using low cost covert channels |
| CN105451197A (en) * | 2014-09-29 | 2016-03-30 | 阿里巴巴集团控股有限公司 | Method and device of sending communication information |
| CN107070613A (en) * | 2017-03-22 | 2017-08-18 | 公安部交通管理科学研究所 | Reliable data transmission method under distributed network environment |
| CN107819599A (en) * | 2016-09-13 | 2018-03-20 | 南京中兴新软件有限责任公司 | Message processing method and device |
| US20180373563A1 (en) * | 2016-03-10 | 2018-12-27 | Alibaba Group Holding Limited | Method and system for processing communication channel |
| CN109194744A (en) * | 2018-09-05 | 2019-01-11 | 上海华测导航技术股份有限公司 | A kind of data transmission method, device, storage medium and monitoring device |
| CN109922050A (en) * | 2019-02-03 | 2019-06-21 | 普信恒业科技发展（北京）有限公司 | A kind of task detection method and device |
| CN110875887A (en) * | 2018-08-31 | 2020-03-10 | 蔚来汽车有限公司 | MQTT protocol-based communication interaction method and communication interaction system |
| CN111258776A (en) * | 2020-01-09 | 2020-06-09 | 上海钧正网络科技有限公司 | Disaster recovery method and device for remote service calling |
| CN112713969A (en) * | 2020-12-30 | 2021-04-27 | 北京字跳网络技术有限公司 | Data transmission method and device and system using same |
-
2021
- 2021-10-19 CN CN202111216434.1A patent/CN113965482B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1145496A2 (en) * | 1999-01-29 | 2001-10-17 | Sun Microsystems, Inc. | A method to monitor and control server applications using low cost covert channels |
| CN105451197A (en) * | 2014-09-29 | 2016-03-30 | 阿里巴巴集团控股有限公司 | Method and device of sending communication information |
| US20180373563A1 (en) * | 2016-03-10 | 2018-12-27 | Alibaba Group Holding Limited | Method and system for processing communication channel |
| CN107819599A (en) * | 2016-09-13 | 2018-03-20 | 南京中兴新软件有限责任公司 | Message processing method and device |
| CN107070613A (en) * | 2017-03-22 | 2017-08-18 | 公安部交通管理科学研究所 | Reliable data transmission method under distributed network environment |
| CN110875887A (en) * | 2018-08-31 | 2020-03-10 | 蔚来汽车有限公司 | MQTT protocol-based communication interaction method and communication interaction system |
| CN109194744A (en) * | 2018-09-05 | 2019-01-11 | 上海华测导航技术股份有限公司 | A kind of data transmission method, device, storage medium and monitoring device |
| CN109922050A (en) * | 2019-02-03 | 2019-06-21 | 普信恒业科技发展（北京）有限公司 | A kind of task detection method and device |
| CN111258776A (en) * | 2020-01-09 | 2020-06-09 | 上海钧正网络科技有限公司 | Disaster recovery method and device for remote service calling |
| CN112713969A (en) * | 2020-12-30 | 2021-04-27 | 北京字跳网络技术有限公司 | Data transmission method and device and system using same |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023160276A1 (en) * | 2022-02-25 | 2023-08-31 | 腾讯科技（深圳）有限公司 | Data processing method and apparatus, computer device and readable storage medium |
| CN114552314A (en) * | 2022-03-02 | 2022-05-27 | 深圳市爱普丰电子有限公司 | Data transmission method based on multi-head data line and data line thereof |
| CN114552314B (en) * | 2022-03-02 | 2024-04-19 | 深圳市爱普丰电子有限公司 | Multi-head data line-based data transmission method and data line thereof |
| CN114697376A (en) * | 2022-03-16 | 2022-07-01 | 浪潮云信息技术股份公司 | Super-large message transmission method and device |
| CN115334000A (en) * | 2022-07-20 | 2022-11-11 | 苏州浪潮智能科技有限公司 | Data transmission method, device, equipment and medium based on multi-control storage system |
| CN115334000B (en) * | 2022-07-20 | 2023-08-11 | 苏州浪潮智能科技有限公司 | Data transmission method, device, equipment and medium based on multi-control storage system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113965482B (en) | 2023-03-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113965482B (en) | Data transmission method and device based on gRPC and storage medium | |
| US20100070588A1 (en) | Reliability for instant messaging based on end point acknowledgements | |
| KR20130082070A (en) | Communication apparatus and communication method | |
| KR20060063652A (en) | Efficient transfer of messages using reliable messaging protocols for web services | |
| CN110413425B (en) | Third-party message callback method, device, server and storage medium | |
| CN114500552B (en) | Cloud edge message reliability transmission method and device under edge computing scene | |
| CN110875887A (en) | MQTT protocol-based communication interaction method and communication interaction system | |
| CN107547505B (en) | Message processing method and device | |
| CN114185582A (en) | Automobile software online upgrading system and method based on QUIC protocol | |
| CN113986501A (en) | Real-time database API (application program interface) uninterrupted calling method, system, storage medium and server | |
| US20140056299A1 (en) | Information processing system, relay device, information processing device, and information processing method | |
| EP1715656A1 (en) | Offering a push service to a wireless device using a push proxy which monitors the coverage state of the device | |
| CN102271067A (en) | network detecting method, device and system | |
| US8370443B2 (en) | Reliable messaging using publish subscribe mechanism | |
| CN110266446B (en) | Method and device for adjusting out-of-order time length based on SACK mode | |
| CN117040692A (en) | Method and device for transmitting service data, electronic equipment and storage medium | |
| EP1662704B1 (en) | Monitoring system, apparatus to be monitored, monitoring apparatus and monitoring method | |
| CN115865886B (en) | HTTP-based cross-network data interaction method and device | |
| CN109067503B (en) | Data retransmission method and device | |
| CN110650204A (en) | Data access method, device, equipment and storage medium of NAS cluster | |
| CN109905209B (en) | Method and device for setting network disorder value | |
| CN116112536A (en) | TCP-serial port communication transparent transmission method and device | |
| CN108880994B (en) | Method and device for retransmitting mails | |
| CN112367265A (en) | Reliable data transmission method and device suitable for narrow-band weak connection network | |
| CN109672686B (en) | Voice data transmission method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |