CN113422739B - Data transmission method, sending end and data transmission system - Google Patents

Data transmission method, sending end and data transmission system Download PDF

Info

Publication number
CN113422739B
CN113422739B CN202110688538.6A CN202110688538A CN113422739B CN 113422739 B CN113422739 B CN 113422739B CN 202110688538 A CN202110688538 A CN 202110688538A CN 113422739 B CN113422739 B CN 113422739B
Authority
CN
China
Prior art keywords
data
transmitted
partition
message queue
application node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110688538.6A
Other languages
Chinese (zh)
Other versions
CN113422739A (en
Inventor
王旭
刘国栋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Bilibili Technology Co Ltd
Original Assignee
Shanghai Bilibili Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Bilibili Technology Co Ltd filed Critical Shanghai Bilibili Technology Co Ltd
Priority to CN202110688538.6A priority Critical patent/CN113422739B/en
Publication of CN113422739A publication Critical patent/CN113422739A/en
Application granted granted Critical
Publication of CN113422739B publication Critical patent/CN113422739B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L49/00—Packet switching elements
    • H04L49/90—Buffering arrangements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00—Routing or path finding of packets in data switching networks
    • H04L45/54—Organization of routing tables
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

The application provides a data transmission method, a sending end and a data transmission system, wherein the data transmission method comprises the following steps: sending the acquired data to be transmitted to a message queue system; under the condition that the transmission of data to be transmitted fails, acquiring a partition routing relation table reported by a message queue system from a preset database; and transmitting the data to be transmitted to the corresponding first application node through a hypertext transmission protocol according to the partition routing relation table. Therefore, when the transmission of the data to be transmitted fails, the sending end can acquire the partition routing relation table from the preset database, can determine the application node to which the data to be transmitted should be transmitted according to the partition routing relation table, directly transmits the data to be transmitted to the corresponding application node through the hypertext transfer protocol, and can ensure that the operation of a user is not influenced when the transmission of the data to be transmitted fails, avoid the data loss, improve the success rate of data transmission and ensure the user experience.

Description

Data transmission method, sending end and data transmission system
Technical Field
The present application relates to the field of internet technologies, and in particular, to a data transmission method. The application also relates to a data sending end, a data transmission system, a computing device and a computer readable storage medium.
Background
With the rapid development of internet technology and computer technology and the massive popularization of PC terminals and smart phones, the amount of information generated in the information era has rapidly increased, and data to be stored, processed and transmitted has also multiplied.
In the prior art, when data transmission is performed, a sending end can send data to be transmitted to a message queue, and transmit the data to each terminal needing to acquire the data through the message queue.
Disclosure of Invention
In view of this, an embodiment of the present application provides a data transmission method. The application also relates to a data sending end, a data transmission system, a computing device and a computer readable storage medium, which are used for solving the problems that the success rate of data transmission is low and the user experience is influenced in the prior art.
According to a first aspect of an embodiment of the present application, a data transmission method is provided, which is applied to a sending end, and includes:
sending the acquired data to be transmitted to a message queue system, wherein the message queue system is used for transmitting the data to be transmitted to an application node;
under the condition that the transmission of the data to be transmitted fails, acquiring a partition routing relation table reported by the message queue system from a preset database, wherein the partition routing relation table is generated by the message queue system according to a data attribute identifier, a partition queue and a corresponding relation between application nodes;
and transmitting the data to be transmitted to the corresponding first application node through a hypertext transmission protocol according to the partition routing relation table.
According to a second aspect of the embodiments of the present application, there is provided a data transmitting end, including:
the sending module is configured to send the obtained data to be transmitted to a message queue system, and the message queue system is used for transmitting the data to be transmitted to an application node;
the acquisition module is configured to acquire a partition routing relation table reported by the message queue system from a preset database under the condition that the transmission of the data to be transmitted fails, wherein the partition routing relation table is generated by the message queue system according to a data attribute identifier, a partition queue and an application node;
and the transmission module is configured to transmit the data to be transmitted to the corresponding first application node through a hypertext transmission protocol according to the partition routing relation table.
According to a third aspect of the embodiments of the present application, a data transmission system is provided, which includes a sending end, a message queue system, a preset database, and an application node;
the sending end is configured to send the acquired data to be transmitted to a message queue system;
the message queue system is configured to store the data to be transmitted into a corresponding first partition queue according to the data attribute identifier of the data to be transmitted, wherein the first partition queue is used for transmitting the data to be transmitted to a first application node, and the first application node is any application node included in the data transmission system; generating a partition routing relation table according to the corresponding relation among the data attribute identification, the partition queue and the application node, and reporting the partition routing relation table to the preset database;
the sending end is further configured to obtain a partition routing relation table reported by the message queue system from the preset database under the condition that the transmission of the data to be transmitted fails, and transmit the data to be transmitted to the first application node through a hypertext transfer protocol according to the partition routing relation table.
According to a fourth aspect of embodiments herein, there is provided a computing device comprising:
a memory and a processor;
the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions to implement the method of:
sending the acquired data to be transmitted to a message queue system, wherein the message queue system is used for transmitting the data to be transmitted to an application node;
under the condition that the transmission of the data to be transmitted fails, acquiring a partition routing relation table reported by the message queue system from a preset database, wherein the partition routing relation table is generated by the message queue system according to a data attribute identifier, a partition queue and a corresponding relation between application nodes;
and transmitting the data to be transmitted to the corresponding first application node through a hypertext transmission protocol according to the partition routing relation table.
According to a fifth aspect of embodiments herein, there is provided a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of any of the data transmission methods.
According to the data transmission method, the obtained data to be transmitted can be sent to a message queue system, and the message queue system is used for transmitting the data to be transmitted to an application node; under the condition that the transmission of the data to be transmitted fails, acquiring a partition routing relation table reported by the message queue system from a preset database, wherein the partition routing relation table is generated by the message queue system according to a data attribute identifier, a partition queue and a corresponding relation between application nodes; and then transmitting the data to be transmitted to the corresponding first application node through a hypertext transmission protocol according to the partition routing relation table.
In this case, after the data to be transmitted is sent to the message queue system, the message queue system can report the partition routing relation table to the preset database in real time, so that when the data to be transmitted fails to be transmitted, the sending end can obtain the partition routing relation table from the preset database, according to the partition routing relation table, the sending end can determine to which application node the data to be transmitted should be transmitted, and directly transmit the data to be transmitted to the corresponding application node through a hypertext transfer protocol, when the data to be transmitted fails to be transmitted, user operation is not affected, data loss is avoided, the success rate of data transmission is improved, and user experience is guaranteed; in addition, the routing relation table can transmit the data to be transmitted to the corresponding first application node according to the original data transmission path, so that the consistency of the data link is ensured, and the possibility of data collision is reduced.
Drawings
Fig. 1 is a flowchart of a data transmission method according to an embodiment of the present application;
fig. 2 is a processing flow diagram of a data transmission method applied to a praise scene according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of a data sending end according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a data transmission system according to an embodiment of the present application;
fig. 5 is a block diagram of a computing device according to an embodiment of the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is capable of implementation in many different ways than those herein set forth and of similar import by those skilled in the art without departing from the spirit of this application and is therefore not limited to the specific implementations disclosed below.
The terminology used in the one or more embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the present application. As used in one or more embodiments of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present application refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It will be understood that, although the terms first, second, etc. may be used herein in one or more embodiments of the present application to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first aspect may be termed a second aspect, and, similarly, a second aspect may be termed a first aspect, without departing from the scope of one or more embodiments of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
First, the noun terms referred to in one or more embodiments of the present application are explained.
Message queue: the method has wide and deep application in large-scale websites, and the message queue can eliminate high concurrency peaks and improve the response speed of the websites.
Kafka: the message queue system is the most popular message queue system of distributed type, partition type, multi-copy type and multi-subscriber, and is widely applied to various large Internet companies and enterprises.
Etcd: an open source item is a highly available distributed key-value store.
HTTP: hypertext transfer protocol, which services can use to obtain data.
Routing (English: Routing): is the activity of transmitting information from a source address to a destination address through an interconnected network.
In the present application, a data transmission method is provided, and the present application relates to a data sending end, a data transmission system, a computing device, and a computer-readable storage medium, which are described in detail in the following embodiments one by one.
Fig. 1 shows a flowchart of a data transmission method provided in an embodiment of the present application, which is applied to a sending end, and specifically includes the following steps:
step 102: and sending the acquired data to be transmitted to a message queue system, wherein the message queue system is used for transmitting the data to be transmitted to an application node.
In practical application, when data transmission is performed, a sending end can send data to be transmitted to a message queue system, and the data to be transmitted is transmitted to each application node through the message queue system.
Therefore, the embodiment of the application provides a data transmission method, which can ensure that user operation is not affected when data to be transmitted fails to be transmitted, avoid data loss, improve the success rate of data transmission, ensure user experience, ensure the consistency of a data link, and reduce the possibility of data collision.
Specifically, the data to be transmitted may refer to data that is generated by a user executing a preset operation and needs to be transmitted to the application node, for example, the data to be transmitted may be approval data of the user. In addition, the message queue system may be a system formed by a plurality of partition queues that receive and store data to be transmitted sent by a sending end and transmit the data to a corresponding application node, where the message queue system includes at least two partition queues, and each partition queue is used to store corresponding data to be transmitted and transmit the data to be transmitted to the corresponding application node. In actual implementation, the message queue system may be Kafka.
In practical application, a user can execute preset operation on a user terminal held by the user, so that corresponding data to be transmitted are generated, a sending end can acquire the data to be transmitted from each user terminal, and then the acquired data to be transmitted are sent to a message queue system. Because the sending end can obtain the data to be transmitted from a plurality of user terminals, the number of the obtained data to be transmitted can be at least one.
For example, the user a performs an approval operation on the user terminal 1 owned by the user a, and at this time, the user terminal 1 generates approval data 1; the user B executes the praise operation on the user terminal 2 held by the user B, and the user terminal 2 generates praise data 2 at the moment; the user C performs the approval operation on the user terminal 3 owned by the user C, and the user terminal 3 generates the approval data 3. The sender may obtain the praise data 1, the praise data 2, and the praise data 3 from the user terminal 1, the user terminal 2, and the user terminal 3, respectively, and then send the praise data 1, the praise data 2, and the praise data 3 to the message queue system.
It should be noted that, after the sending end sends each piece of to-be-transmitted data to the message queue system, the message queue system may store each piece of to-be-transmitted data into a corresponding partition queue according to the data attribute identifier of the to-be-transmitted data, so that each piece of to-be-transmitted data may be transmitted to each corresponding application node through the partition queue in the message queue system.
In specific implementation, the data attribute identifiers of the data to be transmitted, which are acquired by the sending end from different user terminals, are different, so that the message queue system can determine the data attribute identifiers of the data to be transmitted, and then store the data to be transmitted into the corresponding first partition queues according to the data attribute identifiers and preset storage rules. The preset storage rule is a preset rule of a storage relationship between the data attribute identifier and the partition queue, that is, a rule to which the data of each data attribute identifier should be stored correspondingly. That is to say, the message queue system may store the data to be transmitted according to the data attribute identifier of the data to be transmitted and a preset corresponding storage rule.
For example, taking the data to be transmitted as the like, the like can be stored in the 1 st partition queue in the message queue system, the like, in which the user's end number is 1, and the like.
In addition, after each to-be-transmitted data is stored in the corresponding partition queue, the number of queues of the partition queues included in the message queue system and the number of nodes of the application nodes included in the data transmission system can be determined, and then the corresponding first application node is allocated to the first partition queue according to the number of queues, the number of nodes and a preset allocation rule, wherein the first application node is any application node included in the data transmission system. Specifically, the preset allocation rule may refer to a preset rule for indicating which partition queue data should be transmitted to which application node or application nodes, for example, when allocating a corresponding application node to each partition queue, the allocation may be performed based on a load principle, that is, the preset allocation rule may be a load balancing rule.
For example, assuming that the message queue system includes 5 partition queues and there are 5 application nodes in the data transmission system, each application node may allocate 1 partition queue according to the load balancing rule, that is, partition queue 1 is allocated to application node 1, partition queue 2 is allocated to application node 2, and so on. Assuming there are only 2 application nodes in the data transmission system, it is possible to assign partition queue 1 and partition queue 3 to application node 1 and partition queue 2, partition queue 4 and partition queue 5 to application node 2 according to the load balancing rule.
In the embodiment of the application, after the sending end sends the acquired data to be transmitted to the message queue system, the message queue system can store the received data to be transmitted to the corresponding partition queues, and each partition queue is allocated with the corresponding application node, so that each subsequent application node can acquire the data from the corresponding partition queue, and subsequent data processing is performed.
Step 104: and under the condition that the transmission of the data to be transmitted fails, acquiring a partition routing relation table reported by the message queue system from a preset database, wherein the partition routing relation table is generated by the message queue system according to the data attribute identification, the partition queue and the corresponding relation between application nodes.
Specifically, the failure of transmission of the data to be transmitted means that the data to be transmitted cannot be transmitted to the corresponding application node through the message queue system. In practical applications, the failure of data transmission to be transmitted may be caused by a link failure between the sending end and the message queue system, or may also be caused by a failure of the message queue system. Wherein the message queue system failure comprises a partition queue failure included in a message queue system and a link failure between the partition queue and an application node.
In addition, the preset database may be a preset database, and the preset database may receive and store the partition routing relationship table reported by the message queue system, and provide a query obtaining interface for the sending end to obtain the partition routing relationship table reported by the message queue system from the preset database, for example, the preset database may be an Etcd.
In actual implementation, the message queue system may create a partition routing relationship table according to the data attribute identifier of the received data to be transmitted, the corresponding relationship between the partition queue and the application node, and report the partition routing relationship table to the preset database. Specifically, after receiving the data to be transmitted, the message queue system may store the data to be transmitted to the corresponding partition queues according to the data attribute identifiers of the data to be transmitted, and allocate corresponding application nodes to each partition queue, and at this time, the message queue system may create a partition routing relationship table according to the data attribute identifiers, the partition queues, and the corresponding relationships between the application nodes, and report the partition routing relationship table to the preset database. That is, the partition routing relation table may be used to indicate to which partition queue in the message queue the data to be transmitted corresponds, and the partition queue is specifically allocated to the corresponding relation between which application nodes.
In practical application, the message queue system may create a partition routing relationship table based on the received data to be transmitted, and may also set the partition routing relationship table in the message queue system in advance, and report the partition routing relationship table to the preset database, that is, common data attribute identifiers may be determined in advance, corresponding partition queues and application nodes may be determined, and then corresponding partition routing relationship tables may be generated and reported to the preset database. Therefore, after receiving the data to be transmitted sent by the sending end, the message queue system can directly store the data to be transmitted into the corresponding partition queue in the partition routing relation table according to the data attribute identifier of the data to be transmitted, and subsequently provide the data to be transmitted to the corresponding application node.
In addition, after the message queue system generates and reports the partition routing relationship table, if data to be transmitted of the new data attribute identifier is received, or when the application nodes are increased or decreased, the partition routing relationship table can be updated and reported to the preset database in real time.
Illustratively, the message queue system comprises 3 partition queues, and 3 application nodes are provided in total, it is assumed that the partition queue 1 stores the praise data 1 with the data attribute being 1, and the partition queue 1 is allocated to the application node 1; suppose that the partition queue 2 stores praise data 2 with a data attribute of 2, and the partition queue 2 is allocated to the application node 2; assume that the partition queue 3 stores therein like data 3 whose data attribute is identified to be 3, and the partition queue 3 is assigned to the application node 3. Meanwhile, the message queue system reports a partition routing relation table shown in the following table 1 to a preset database. Assuming that the application node 4 and the application node 5 are newly added after a period of time, and the partition queue 1 is allocated to the application node 4, and the partition queue 2 is allocated to the application node 5, an updated partition routing relationship table as shown in the following table 2 can be obtained.
Table 1 partitioned routing relation table
Data attribute identification Partitioned queue Application node
1 1 1
2 2 2
3 3 3
Table 2 updated partition routing relation table
Data attribute identification Partitioned queue Application node
1 1 1、4
2 2 2、5
3 3 3
In the embodiment of the application, under the condition that the data to be transmitted fails to be transmitted, it is described that the data to be transmitted is not successfully transmitted to the message queue system, that is, the data to be transmitted cannot be transmitted to the corresponding application node through each partition queue included in the message queue system, at this time, the transmitting end can acquire the partition routing relation table reported by the message queue system from the preset database, so that which data to be transmitted should be transmitted to which application node is determined, it is convenient for data transmission to be subsequently performed by adopting other modes, and the success rate of data transmission and the consistency of data paths are ensured.
In an optional implementation manner of this embodiment, whether an exception occurs in a transmission process may be determined in different manners, so as to determine whether transmission of data to be transmitted fails, that is, before acquiring a partition routing relationship table reported by the message queue system from a preset database under a condition that transmission of the data to be transmitted fails, the method further includes:
determining that the data to be transmitted fails to be transmitted under the condition of receiving the abnormal message returned by the message queue system; alternatively, the first and second liquid crystal display panels may be,
and determining that the transmission of the data to be transmitted fails under the condition that a transmission success message returned by the message queue system is not received within a first preset time length.
Specifically, the first preset time period is a preset time period, such as 5 minutes, 10 minutes, 20 minutes, and the like.
In practical implementation, in a possible implementation manner, when the message queue system is abnormal, the message queue system may return an abnormal message to the sending end, and if the sending end receives the abnormal message, it indicates that the message queue system is abnormal, and the message queue system cannot successfully implement data transmission, and at this time, it may be determined that the data transmission to be transmitted fails.
In another possible implementation manner, after the message queue system successfully transmits the data to be transmitted to the application node, a transmission success message may also be returned to the sending end, and if a link between the sending end and the message queue system is abnormal or the message queue system is abnormal, the sending end cannot receive the transmission success message returned by the message queue system. That is, if the transmitting end does not receive a transmission success message returned by the message queue system within the first preset duration, it indicates that the link between the transmitting end and the message queue system is abnormal, or the message queue system is abnormal, and data transmission cannot be successfully realized, and at this time, it may be determined that the data transmission to be transmitted fails.
It should be noted that, when the transmitting end does not determine that the transmission of the data to be transmitted fails, it is described that the partition queue included in the message queue system can transmit the data to be transmitted to the corresponding application node, and at this time, the data transmission is normally performed through the partition queue included in the message queue system; and if the transmitting end determines that the data to be transmitted fails to be transmitted, the data transmission process is abnormal, the data to be transmitted cannot be transmitted to the corresponding application node through the partition queue included in the message queue system, and at this moment, the transmitting end can acquire the partition routing relation table reported by the message queue system from the preset database, so that the data to be transmitted can be transmitted to the corresponding application node in other ways in the follow-up process.
In specific implementation, when the sending end obtains the partition routing relation table reported by the message queue system from the preset database, the sending end can obtain the partition routing relation table reported latest by the message queue system at the current time.
Along the above example, if the sending end does not determine that the data to be transmitted fails to be transmitted, it is indicated that data transmission can be achieved through the message queue system, at this time, the partition queue 1 may transmit the stored approval data 1 to the application node 1 and the application node 4, the partition queue 2 may transmit the stored approval data 2 to the application node 2 and the application node 5, and the partition queue 3 may transmit the stored approval data 3 to the application node 3. If the sending end determines that the data transmission to be transmitted fails, it indicates that the data transmission cannot be realized through the message queue system, and at this time, the sending end may obtain the partition routing relationship table shown in table 2 from a preset database.
In the embodiment of the application, when the data transmission to be transmitted fails, the transmitting end can timely know that the data transmission process is abnormal, and the data transmission cannot be realized, so that the transmitting end can timely adopt other transmission means, and the success rate of the data transmission is ensured.
Step 106: and transmitting the data to be transmitted to the corresponding first application node through a hypertext transmission protocol according to the partition routing relation table.
It should be noted that the partition routing relationship table may indicate which partition queue in the message queue the to-be-transmitted data corresponds to, and the partition queue is specifically allocated to a corresponding relationship between which application node, and according to the partition routing relationship table, the sending end may know to which application node the to-be-transmitted data should be originally transmitted, so that the to-be-transmitted data is actually pushed through a hypertext transfer protocol (HTTP).
In an optional implementation manner of this embodiment, the transmitting the data to be transmitted to the corresponding first application node through a hypertext transfer protocol according to the partition routing relation table may specifically be implemented as follows:
determining a data attribute identifier of the data to be transmitted;
determining a first application node corresponding to the data attribute identification of the data to be transmitted according to the partition routing relation table;
and transmitting the data to be transmitted to the first application node through a hypertext transfer protocol.
It should be noted that, according to the partition routing relationship table, it may be determined which attribute of the to-be-transmitted data should be stored in which partition queue in the message queue system and should be transmitted to which application node, so that after the sending end acquires the partition routing relationship table, the data attribute identifier of the current to-be-transmitted data may be determined first, and then the first application node corresponding to the data attribute identifier is determined by looking up in the partition routing relationship table, where the first application node is the application node to which the to-be-transmitted data should be originally transmitted through the message queue system, and at this time, the to-be-transmitted data may be transmitted to the first application node through a hypertext transfer protocol.
Along the application example, the data to be transmitted are the like data 1, the like data 2 and the like data 3, the partition routing relationship table obtained by the sending end is shown in the table 2, the sending end can determine that the data attribute identifier of the like data 1 is 1, and according to the partition routing relationship table shown in the table 2, the first application node corresponding to the data attribute identifier 1 is determined to be the application node 1 and the application node 4; determining that the data attribute identifier of the compliment data 2 is 2, and determining that the first application node corresponding to the data attribute identifier 2 is an application node 2 and an application node 5 according to the partition routing relation table shown in the table 2; and determining that the data attribute identifier of the praise data 3 is 3, and determining that the first application node corresponding to the data attribute identifier 3 is the application node 3 according to the partition routing relation table shown in the table 2. At this time, the sending end may push the like data 1 to the application node 1 and the application node 4 through HTTP, push the like data 2 to the application node 2 and the application node 5 through HTTP, and push the like data 3 to the application node 3 through HTTP.
In an optional implementation manner of this embodiment, when an application node normally obtains corresponding to-be-transmitted data from a partition queue in a message queue system, the application node may further expose an HTTP interface to the outside and monitor the data, so that the to-be-transmitted data is transmitted to the first application node through a hypertext transfer protocol, and a specific implementation process may be as follows:
and sending the data to be transmitted to a hypertext transfer protocol interface of the first application node, wherein the hypertext transfer protocol interface is an interface which is exposed and monitored by the first application node.
It should be noted that each first application node exposes its own HTTP interface to the outside, so that the sending end can obtain the HTTP interface exposed to the outside of the first application node after determining which first application node the data to be transmitted is to be pushed to.
According to the method and the device, a sending end can determine which attribute of data to be transmitted, the data to be transmitted should be stored in which partition queue in a message queue system and should be transmitted to which application node, so that after the sending end obtains the partition routing relation table, the data to be transmitted can be directly transmitted to the corresponding application node through a hypertext transfer protocol, when the data to be transmitted fails to be transmitted, user operation is not affected, data loss is avoided, the success rate of data transmission is improved, and user experience is guaranteed; in addition, the data to be transmitted is transmitted to the corresponding first application node according to the original data transmission path through the routing relation table, so that the consistency of the data link is ensured, and the possibility of data collision is reduced.
In an optional implementation manner of this embodiment, in a case that transmission of data to be transmitted fails, the data to be transmitted may be transmitted to a corresponding first application node through a hypertext transfer protocol, but after an exception in a data transmission process is repaired, data transmission may be continuously achieved through a message queue system, so that a sending end may further determine and record whether the exception in the data transmission process is repaired, that is, according to the partition routing relation table, after the data to be transmitted is transmitted to the corresponding first application node through the hypertext transfer protocol, the method further includes:
under the condition of receiving a recovery message returned by the message queue system, recording the data transmission state of the message queue system as normal transmission; alternatively, the first and second electrodes may be,
and sending a connection establishing request to the message queue system every second preset time, establishing connection with the message queue system under the condition of receiving a response message returned by the message queue system, and recording the data transmission state of the message queue system as normal transmission.
Specifically, the second preset time period is a preset time period, such as 5 minutes, 10 minutes, 15 minutes, and the like. The second preset time period may be the same as or different from the first preset time period.
It should be noted that, in a possible implementation manner, if the transmission failure of the data to be transmitted is caused by an abnormality of the message queue system, the message queue system may return a recovery message to the sending end after recovering to normal, and the sending end records the data transmission state of the message queue system as normal transmission when receiving the recovery message. The data transmission state of the message queue system is normal transmission, which means that the message queue system can be used for transmitting data to be transmitted to the corresponding application node subsequently.
In another possible implementation manner, if the transmission failure of the data to be transmitted is caused by an abnormal link between the sending end and the message queue system, the sending end may send a connection establishment request to the message queue system every second preset time period, and if the link between the sending end and the message queue system returns to normal, the sending end may receive a response message returned by the message queue system. That is, if the sending end receives a response message returned by the message queue system, it indicates that the link between the sending end and the message queue system is recovered to normal, and at this time, the data transmission state of the message queue system may be recorded as normal transmission.
In an optional implementation manner of this embodiment, in the case that transmission of data to be transmitted fails, data to be transmitted that is not successfully transmitted may be pushed to a corresponding first application node through HTTP according to an obtained partition routing relation table and an original transmission path. Then, the sending end may also obtain new data to be transmitted, and at this time, may query whether the data transmission state of the message queue system is normal transmission, so as to determine a transmission mode of the new data to be transmitted, that is, after the data to be transmitted is transmitted to the corresponding first application node through the hypertext transfer protocol according to the partition routing relation table, the method further includes:
under the condition of acquiring new data to be transmitted, inquiring the data transmission state of the message queue system;
if the data transmission state is normal transmission, sending the new data to be transmitted to the message queue system, and transmitting the new data to be transmitted to a corresponding second application node through the message queue system;
and if the data transmission state is not normal transmission, determining whether the new data to be transmitted has a corresponding second application node according to the partition routing relation table, and if so, transmitting the new data to be transmitted to the second application node through a hypertext transfer protocol.
Specifically, the new data to be transmitted is the data to be transmitted newly acquired by the sending end. In practical application, when a sending end acquires new data to be transmitted, it is indicated that the new data to be transmitted needs to be transmitted to a corresponding application node, at this time, a data transmission state of the message queue system may be queried, if the data transmission state is normal transmission, it is indicated that the new data to be transmitted may be transmitted to the corresponding application node by the message queue system, and if the data transmission state is not normal transmission, it is indicated that the new data to be transmitted may not be transmitted to the corresponding application node by the message queue system.
In a possible implementation manner, if the data transmission state is normal transmission, it indicates that the new data to be transmitted may be transmitted to the corresponding application node through the message queue system, and at this time, the new data to be transmitted may be sent to the message queue system, and the message queue system may store the new data to be transmitted in the corresponding second partition queue and allocate the corresponding second application node to the second partition queue, so that the new data to be transmitted is transmitted to the second application node through the second partition queue.
In the above example, the new data to be transmitted is the like data 4 (the data attribute is identified as 1) and the like data 5 (the data attribute is identified as 3), and assuming that the message queue system returns to normal, the like data 4 and the like data 5 may be sent to the message queue system, and the message queue system stores the like data 4 into the partition queue 1 and stores the like data 5 into the partition queue 3 according to the preset storage rule. Subsequently, the approval data 4 may be transmitted to the application node 1 and the application node 4 through the partition queue 1, and the approval data 5 may be transmitted to the application node 3 through the partition queue 3.
In another possible implementation manner, if the data transmission state is not normal transmission, it indicates that the new data to be transmitted cannot be transmitted to the corresponding application node through the message queue system, so that the new data to be transmitted may not be sent to the message queue system at this time, and the new data to be transmitted is continuously transmitted to the corresponding application node according to the obtained partition routing relationship table.
It should be noted that if the data attribute identifier of the new data to be transmitted is the same as any data attribute identifier of the partition routing relationship table, the obtained partition routing relationship table has a second application node corresponding to the data attribute identifier of the new data to be transmitted, and at this time, the new data to be transmitted can be directly transmitted to the second application node through the hypertext transfer protocol; if the data attribute identification of the new data to be transmitted is different from any data attribute identification of the partition routing relationship table, the obtained partition routing relationship table does not have a second application node corresponding to the data attribute identification of the new data to be transmitted, at this moment, a data transmission failure can be returned, and the new data to be transmitted can also be transmitted to any application node through HTTP.
Along the above example, the new data to be transmitted is the approval data 4 (the data attribute identifier is 1) and the approval data 5 (the data attribute identifier is 3), and if the data transmission state of the message queue system is not normal, the second application nodes corresponding to the approval data 4 are determined to be the application node 1 and the application node 4 according to the partition routing relationship table shown in the table 2, and the second application node corresponding to the approval data 5 is the application node 3. The sending end can push the approval data 4 to the application node 1 and the application node 4 through HTTP, and push the approval data 5 to the application node 3 through HTTP.
According to the data transmission method, after the data to be transmitted is sent to the message queue system, the message queue system can report the partition routing relation table to the preset database in real time, so that when the data to be transmitted fails to be transmitted, the sending end can obtain the partition routing relation table from the preset database, and according to the partition routing relation table, the sending end can determine to which application node the data to be transmitted should be transmitted, and directly transmits the data to be transmitted to the corresponding application node through a hypertext transmission protocol; in addition, the data to be transmitted is transmitted to the corresponding first application node according to the original data transmission path through the routing relation table, so that the consistency of the data link is ensured, and the possibility of data collision is reduced.
The data transmission method provided by the present application is further described below with reference to fig. 2 by taking an application of the data transmission method in a praise scene as an example. Fig. 2 shows a processing flow chart of a data transmission method applied to a praise scene according to an embodiment of the present application, which specifically includes the following steps:
step 202: and the sending end acquires the praise data from each user terminal and sends the acquired praise data to the kafka.
In an example, the sending end acquires the praise data 1 sent with the tail number 1, the praise data 2 sent with the tail number 2, and the praise data 3 sent with the tail number 3, and sends the praise data 1, the praise data 2, and the praise data 3 to the kafka.
Step 204: kafka stores each piece of praise data into a corresponding partition queue according to the data attribute identification of the praise data and a preset storage rule, determines the number of queues of the partition queues included in a message queue system and the number of nodes of application nodes included in a data transmission system, and allocates the corresponding application nodes to each partition queue according to the number of queues, the number of nodes and a preset allocation rule.
For example, the praise data 1 sent with the tail number of 1 is stored in the 1 st partition queue in kafka, the praise data 2 sent with the tail number of 2 is stored in the 2 nd partition queue in kafka, and the praise data 3 sent with the tail number of 3 is stored in the 3 rd partition queue in kafka. Since the number of the partition queues included in the message queue system is 3, the number of the nodes of the application nodes included in the data transmission system is also 3, and a preset allocation rule is assumed as a load balancing rule, that is, one partition queue is allocated to one application node, at this time, the 1 st partition queue may be allocated to the application node 1, the 2 nd partition queue may be allocated to the application node 2, and the 3 rd partition queue may be allocated to the application node 3.
Step 206: and the kafka creates a partition routing relation table according to the data attribute identification, the corresponding relation table between the partition queue and the application node, and reports the partition routing relation table to the Etcd.
Step 208: and under the condition that the point approves data transmission failure, the transmitting end acquires the partition routing relation table reported by the kafka from the Etcd.
Step 210: and the sending end determines an application node corresponding to each praise data according to the partition routing relation table, and transmits the praise data to the corresponding application node through HTTP.
For example, according to the partition routing relation table, it is determined that the application node corresponding to the like data 1 is the application node 1, the application node corresponding to the like data 2 is the application node 2, and the application node corresponding to the like data 3 is the application node 3. The sending end pushes the praise data 1 to the application node 1 through HTTP, pushes the praise data 2 to the application node 2, and pushes the praise data 3 to the application node 3.
According to the data transmission method, after the data to be transmitted is sent to the message queue system, the message queue system can report the partition routing relation table to the preset database in real time, so that when the data to be transmitted fails to be transmitted, the sending end can obtain the partition routing relation table from the preset database, and according to the partition routing relation table, the sending end can determine to which application node the data to be transmitted should be transmitted, and directly transmits the data to be transmitted to the corresponding application node through a hypertext transmission protocol; in addition, the data to be transmitted is transmitted to the corresponding first application node according to the original data transmission path through the routing relation table, so that the consistency of the data link is ensured, and the possibility of data collision is reduced.
The foregoing is an exemplary scheme of applying the data transmission of this embodiment to the praise scene. It should be noted that, the technical solution of the data transmission method applied in the praise scene belongs to the same concept as the technical solution of the data transmission method shown in fig. 1, and details of the technical solution of the data transmission method applied in the praise scene, which are not described in detail, can be referred to the description of the technical solution of the data transmission method shown in fig. 1.
Corresponding to the foregoing method embodiment, the present application further provides a data sending end embodiment, and fig. 3 illustrates a schematic structural diagram of a data sending end provided in an embodiment of the present application. As shown in fig. 3, the data transmitting end includes:
a sending module 302, configured to send the obtained data to be transmitted to a message queue system, where the message queue system is used to transmit the data to be transmitted to an application node;
an obtaining module 304, configured to obtain, from a preset database, a partition routing relationship table reported by the message queue system when transmission of the to-be-transmitted data fails, where the partition routing relationship table is generated by the message queue system according to a data attribute identifier, a partition queue, and a correspondence between application nodes;
a transmission module 306 configured to transmit the data to be transmitted to the corresponding first application node through a hypertext transfer protocol according to the partition routing relation table.
Optionally, the transmission module 306 is further configured to:
determining a data attribute identifier of the data to be transmitted;
determining a first application node corresponding to the data attribute identification of the data to be transmitted according to the partition routing relation table;
and transmitting the data to be transmitted to the first application node through a hypertext transfer protocol.
Optionally, the transmission module 306 is further configured to:
and sending the data to be transmitted to a hypertext transfer protocol interface of the first application node, wherein the hypertext transfer protocol interface is an interface which is exposed and monitored by the first application node.
Optionally, the obtaining module 304 is further configured to:
determining that the transmission of the data to be transmitted fails under the condition of receiving the abnormal message returned by the message queue system; alternatively, the first and second electrodes may be,
and determining that the transmission of the data to be transmitted fails under the condition that a transmission success message returned by the message queue system is not received within a first preset time length.
Optionally, the data sending end further includes a recording module configured to:
under the condition of receiving a recovery message returned by the message queue system, recording the data transmission state of the message queue system as normal transmission; alternatively, the first and second electrodes may be,
and sending a connection establishing request to the message queue system every second preset time, establishing connection with the message queue system under the condition of receiving a response message returned by the message queue system, and recording the data transmission state of the message queue system as normal transmission.
Optionally, the data sending end further includes a query module configured to:
under the condition of acquiring new data to be transmitted, inquiring the data transmission state of the message queue system;
if the data transmission state is normal transmission, sending the new data to be transmitted to the message queue system, and transmitting the new data to be transmitted to a corresponding second application node through the message queue system;
and if the data transmission state is not normal transmission, determining whether the new data to be transmitted has a corresponding second application node according to the partition routing relation table, and if so, transmitting the new data to be transmitted to the second application node through a hypertext transfer protocol.
According to the method and the device, after the data to be transmitted is sent to the message queue system, the message queue system can report the partition routing relation table to the preset database in real time, so that when the data to be transmitted fails to be transmitted, the sending end can obtain the partition routing relation table from the preset database, according to the partition routing relation table, the sending end can determine the application node to which the data to be transmitted should be transmitted, the data to be transmitted is directly transmitted to the corresponding application node through a hypertext transmission protocol, when the data to be transmitted fails to be transmitted, user operation is not affected, data loss is avoided, the success rate of data transmission is improved, and user experience is guaranteed; in addition, the data to be transmitted is transmitted to the corresponding first application node through the routing relation table according to the original data transmission path, so that the consistency of the data link is ensured, and the possibility of data collision is reduced.
The foregoing is an exemplary scheme of a data sending end according to this embodiment. It should be noted that the technical solution of the data sending end and the technical solution of the data transmission method belong to the same concept, and details that are not described in detail in the technical solution of the data sending end can be referred to the description of the technical solution of the data transmission method.
Corresponding to the above method embodiment, the present application further provides an embodiment of a data transmission system, and fig. 4 shows a schematic structural diagram of the data transmission system provided in the embodiment of the present application. As shown in fig. 4, the data transmission system includes a sending end 402, a message queue system 404, a preset database 406, and an application node 408;
the sending end 402 is configured to send the obtained data to be transmitted to the message queue system 404;
the message queue system 404 is configured to store the data to be transmitted into a corresponding first partition queue according to the data attribute identifier of the data to be transmitted, where the first partition queue is used to transmit the data to be transmitted to a first application node 408, and the first application node 408 is any application node included in the data transmission system; generating a partition routing relation table according to the corresponding relation among the data attribute identifier, the partition queue and the application node, and reporting the partition routing relation table to the preset database 406;
the sending end 402 is further configured to, when the transmission of the data to be transmitted fails, obtain a partition routing relationship table reported by the message queue system 404 from the preset database 406, and transmit the data to be transmitted to the first application node 408 through a hypertext transfer protocol according to the partition routing relationship table.
Optionally, the message queue system 404 is further configured to:
determining a data attribute identifier of the data to be transmitted;
and storing the data to be transmitted into a corresponding first partition queue according to the data attribute identification and a preset storage rule.
Optionally, the message queue system 404 is further configured to:
determining the queue number of a partition queue included in the message queue system and the node number of an application node included in the data transmission system;
and distributing corresponding first application nodes for the first partition queue according to the number of the queues, the number of the nodes and a preset distribution rule.
Optionally, the first application node 408 is configured to:
acquiring the data to be transmitted from the first partition queue;
and exposing and monitoring a hypertext transfer protocol interface to the outside.
Optionally, the transmitting end 402 is further configured to:
determining a data attribute identifier of the data to be transmitted;
determining a first application node 408 corresponding to the data attribute identifier of the data to be transmitted according to the partition routing relation table;
the data to be transmitted is transmitted to the first application node 408 via a hypertext transfer protocol.
Optionally, the transmitting end 402 is further configured to:
and sending the data to be transmitted to a hypertext transfer protocol interface of the first application node 408, where the hypertext transfer protocol interface is an interface that is exposed and monitored by the first application node 408.
Optionally, the transmitting end 402 is further configured to:
determining that the data to be transmitted fails to be transmitted under the condition of receiving the abnormal message returned by the message queue system 404; alternatively, the first and second electrodes may be,
and determining that the transmission of the data to be transmitted fails when the transmission success message returned by the message queue system 404 is not received within a first preset time length.
Optionally, the transmitting end 402 is further configured to:
under the condition of receiving a recovery message returned by the message queue system 404, recording that the data transmission state of the message queue system 404 is normal transmission; alternatively, the first and second electrodes may be,
sending a connection establishment request to the message queue system 404 every second preset time interval, establishing connection with the message queue system 404 under the condition of receiving a response message returned by the message queue system 404, and recording that the data transmission state of the message queue system 404 is normal transmission.
According to the method and the device, after the data to be transmitted is sent to the message queue system, the message queue system can report the partition routing relation table to the preset database in real time, so that when the data to be transmitted fails to be transmitted, the sending end can obtain the partition routing relation table from the preset database, according to the partition routing relation table, the sending end can determine the application node to which the data to be transmitted should be transmitted, the data to be transmitted is directly transmitted to the corresponding application node through a hypertext transmission protocol, when the data to be transmitted fails to be transmitted, user operation is not affected, data loss is avoided, the success rate of data transmission is improved, and user experience is guaranteed; in addition, the data to be transmitted is transmitted to the corresponding first application node according to the original data transmission path through the routing relation table, so that the consistency of the data link is ensured, and the possibility of data collision is reduced.
The above is a schematic scheme of a data transmission system of this embodiment. It should be noted that the technical solution of the data transmission system and the technical solution of the data transmission method belong to the same concept, and details that are not described in detail in the technical solution of the data transmission system can be referred to the description of the technical solution of the data transmission method.
Fig. 5 illustrates a block diagram of a computing device 500 provided according to an embodiment of the present application. The components of the computing device 500 include, but are not limited to, a memory 510 and a processor 520. Processor 520 is coupled to memory 510 via bus 530, and database 550 is used to store data.
Computing device 500 also includes access device 540, access device 540 enabling computing device 500 to communicate via one or more networks 560. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. The access device 540 may include one or more of any type of network interface, e.g., a Network Interface Card (NIC), wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.
In one embodiment of the application, the above-described components of computing device 500 and other components not shown in FIG. 5 may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device architecture shown in FIG. 5 is for purposes of example only and is not limiting as to the scope of the present application. Those skilled in the art may add or replace other components as desired.
Computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smartphone), wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 500 may also be a mobile or stationary server.
Wherein processor 520 is configured to execute the following computer-executable instructions:
sending the acquired data to be transmitted to a message queue system, wherein the message queue system is used for transmitting the data to be transmitted to an application node;
under the condition that the transmission of the data to be transmitted fails, acquiring a partition routing relation table reported by the message queue system from a preset database, wherein the partition routing relation table is generated by the message queue system according to data attribute identification, a partition queue and a corresponding relation between application nodes;
and transmitting the data to be transmitted to the corresponding first application node through a hypertext transmission protocol according to the partition routing relation table.
The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the data transmission method belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the data transmission method.
An embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions, when executed by a processor, for implementing the steps of any one of the data transmission methods described above.
The above is an illustrative scheme of a computer-readable storage medium of the embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the data transmission method, and for details that are not described in detail in the technical solution of the storage medium, reference may be made to the description of the technical solution of the data transmission method.
The foregoing description of specific embodiments of the present application has been presented. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
It should be noted that, for the sake of simplicity, the above-mentioned method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present application is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.
In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to the related descriptions of other embodiments.
The preferred embodiments of the present application disclosed above are intended only to aid in the explanation of the application. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the application and its practical applications, to thereby enable others skilled in the art to best understand and utilize the application. The application is limited only by the claims and their full scope and equivalents.

Claims (13)

1. A data transmission method is applied to a sending end and comprises the following steps:
sending the acquired data to be transmitted to a message queue system, wherein the message queue system comprises at least two partition queues, and the partition queues are used for transmitting the data to be transmitted to corresponding application nodes;
under the condition that the transmission of the data to be transmitted fails, acquiring a partition routing relation table reported by the message queue system from a preset database, wherein the partition routing relation table is generated by the message queue system according to a data attribute identifier, a partition queue and a corresponding relation between application nodes;
and transmitting the data to be transmitted to the corresponding first application node through a hypertext transmission protocol according to the partition routing relation table.
2. The data transmission method according to claim 1, wherein the transmitting the data to be transmitted to the corresponding first application node through a hypertext transfer protocol according to the partition routing relation table includes:
determining a data attribute identifier of the data to be transmitted;
determining a first application node corresponding to the data attribute identification of the data to be transmitted according to the partition routing relation table;
and transmitting the data to be transmitted to the first application node through a hypertext transfer protocol.
3. The method according to claim 2, wherein the transmitting the data to be transmitted to the first application node via a hypertext transfer protocol includes:
and sending the data to be transmitted to a hypertext transfer protocol interface of the first application node, wherein the hypertext transfer protocol interface is an interface which is exposed and monitored by the first application node.
4. The data transmission method according to any one of claims 1 to 3, wherein before acquiring the partition routing relationship table reported by the message queue system from a preset database when the transmission of the data to be transmitted fails, the method further includes:
determining that the transmission of the data to be transmitted fails under the condition of receiving the abnormal message returned by the message queue system; alternatively, the first and second electrodes may be,
and determining that the transmission of the data to be transmitted fails under the condition that a transmission success message returned by the message queue system is not received within a first preset time length.
5. The data transmission method according to any one of claims 1 to 3, wherein after the transmitting the data to be transmitted to the corresponding first application node through the hypertext transfer protocol according to the partition routing relation table, the method further includes:
under the condition of receiving a recovery message returned by the message queue system, recording the data transmission state of the message queue system as normal transmission; alternatively, the first and second electrodes may be,
and sending a connection establishment request to the message queue system every other second preset time, establishing connection with the message queue system under the condition of receiving a response message returned by the message queue system, and recording the data transmission state of the message queue system as normal transmission.
6. The data transmission method according to claim 5, wherein after transmitting the data to be transmitted to the corresponding first application node through the hypertext transfer protocol according to the partition routing relation table, the method further comprises:
under the condition of acquiring new data to be transmitted, inquiring the data transmission state of the message queue system;
if the data transmission state is normal transmission, sending the new data to be transmitted to the message queue system, and transmitting the new data to be transmitted to a corresponding second application node through the message queue system;
and if the data transmission state is not normal transmission, determining whether the new data to be transmitted has a corresponding second application node according to the partition routing relation table, and if so, transmitting the new data to be transmitted to the second application node through a hypertext transfer protocol.
7. A data transmitting end, comprising:
the sending module is configured to send the obtained data to be transmitted to a message queue system, the message queue system comprises at least two partition queues, and the partition queues are used for transmitting the data to be transmitted to corresponding application nodes;
the acquisition module is configured to acquire a partition routing relation table reported by the message queue system from a preset database under the condition that the transmission of the data to be transmitted fails, wherein the partition routing relation table is generated by the message queue system according to a data attribute identifier, a partition queue and an application node;
and the transmission module is configured to transmit the data to be transmitted to the corresponding first application node through a hypertext transfer protocol according to the partition routing relation table.
8. A data transmission system is characterized by comprising a sending end, a message queue system comprising at least two partition queues, a preset database and an application node;
the sending end is configured to send the acquired data to be transmitted to the message queue system;
the message queue system is configured to store the data to be transmitted into a corresponding first partition queue according to the data attribute identifier of the data to be transmitted, the first partition queue is used for transmitting the data to be transmitted to a first application node, and the first application node is any application node included in the data transmission system; generating a partition routing relation table according to the corresponding relation among the data attribute identification, the partition queue and the application node, and reporting the partition routing relation table to the preset database;
the sending end is further configured to obtain a partition routing relation table reported by the message queue system from the preset database under the condition that the transmission of the data to be transmitted fails, and transmit the data to be transmitted to the first application node through a hypertext transmission protocol according to the partition routing relation table.
9. The data transmission system of claim 8, wherein the message queue system is further configured to:
determining a data attribute identifier of the data to be transmitted;
and storing the data to be transmitted into a corresponding first partition queue according to the data attribute identification and a preset storage rule.
10. The data transmission system of claim 9, wherein the message queue system is further configured to:
determining the queue number of the partition queues included in the message queue system and the node number of the application nodes included in the data transmission system;
and distributing corresponding first application nodes for the first partition queue according to the number of the queues, the number of the nodes and a preset distribution rule.
11. A data transmission system according to any one of claims 8-10, characterized in that the first application node is configured to:
acquiring the data to be transmitted from the first partition queue;
and exposing and monitoring a hypertext transfer protocol interface to the outside.
12. A computing device, comprising:
a memory and a processor;
the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions to implement the method of:
sending the acquired data to be transmitted to a message queue system, wherein the message queue system comprises at least two partition queues, and the partition queues are used for transmitting the data to be transmitted to corresponding application nodes;
under the condition that the transmission of the data to be transmitted fails, acquiring a partition routing relation table reported by the message queue system from a preset database, wherein the partition routing relation table is generated by the message queue system according to a data attribute identifier, a partition queue and a corresponding relation between application nodes;
and transmitting the data to be transmitted to the corresponding first application node through a hypertext transmission protocol according to the partition routing relation table.
13. A computer-readable storage medium, having stored thereon computer-executable instructions which, when executed by a processor, carry out the steps of the data transmission method according to any one of claims 1 to 6.
CN202110688538.6A 2021-06-21 2021-06-21 Data transmission method, sending end and data transmission system Active CN113422739B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110688538.6A CN113422739B (en) 2021-06-21 2021-06-21 Data transmission method, sending end and data transmission system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110688538.6A CN113422739B (en) 2021-06-21 2021-06-21 Data transmission method, sending end and data transmission system

Publications (2)

Publication Number Publication Date
CN113422739A CN113422739A (en) 2021-09-21
CN113422739B true CN113422739B (en) 2022-09-06

Family

ID=77789703

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110688538.6A Active CN113422739B (en) 2021-06-21 2021-06-21 Data transmission method, sending end and data transmission system

Country Status (1)

Country Link
CN (1) CN113422739B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114900868B (en) * 2022-05-12 2022-11-25 哈尔滨工业大学 Unmanned aerial vehicle ad hoc network routing protocol data transmission method based on energy optimization

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2013086455A1 (en) * 2011-12-07 2013-06-13 Seven Networks, Inc. Flexible and dynamic integration schemas of a traffic management system with various network operators for network traffic alleviation
CN103856392B (en) * 2013-06-26 2017-05-24 上海携程商务有限公司 Message push method, outgoing server using message push method and outgoing server system
US10432448B2 (en) * 2017-07-06 2019-10-01 Sap Se Systems and methods for stream-based, protocol-agnostic messaging
CN108228363A (en) * 2017-12-22 2018-06-29 中国银联股份有限公司 A kind of message method and device
CN109274466B (en) * 2018-08-07 2022-02-22 中国人民银行清算总中心 Middleware message transmission method and system
CN109120713A (en) * 2018-09-21 2019-01-01 中国建设银行股份有限公司 Message sends the processing method and system, device and storage medium of failure
CN109271265B (en) * 2018-09-29 2023-09-15 平安科技（深圳）有限公司 Request processing method, device, equipment and storage medium based on message queue
CN109660607B (en) * 2018-12-05 2021-08-27 北京金山云网络技术有限公司 Service request distribution method, service request receiving method, service request distribution device, service request receiving device and server cluster
CN110740103A (en) * 2019-09-02 2020-01-31 深圳壹账通智能科技有限公司 Service request processing method and device, computer equipment and storage medium
CN110830366A (en) * 2019-11-14 2020-02-21 北京京航计算通讯研究所 Military field message pushing system
CN112532528B (en) * 2020-11-30 2023-04-25 北京百度网讯科技有限公司 Message routing method and device for rule engine

Also Published As

Publication number Publication date
CN113422739A (en) 2021-09-21

Similar Documents

Publication Publication Date Title
CN109949111B (en) Electronic bill identification distribution method, electronic bill generation method, device and system
CN111935315B (en) Block synchronization method and device
CN110300050A (en) Information push method, device, computer equipment and storage medium
CN107181821A (en) A kind of information push method and device based on SSE specifications
CN110505618B (en) Service processing method and server
CN108600316B (en) Data management method, system and equipment of cloud storage service
CN113285884B (en) Flow control method and system
CN113938516A (en) Method and system for synchronously realizing transaction processing of heterogeneous system
CN113422739B (en) Data transmission method, sending end and data transmission system
CN106330833A (en) Communication method based on ICAP (Internet Content Adaptation Protocol), client and server
CN113434312A (en) Data blood relationship processing method and device
CN114885020A (en) Data transmission system and method
CN105912477B (en) A kind of method, apparatus and system that catalogue is read
CN115412592A (en) Service processing system and method
CN106899652B (en) Method and device for pushing service processing result
CN112835766A (en) Application monitoring method and device
CN112612848A (en) Method, device and system for real-time data synchronization, electronic equipment and storage medium
CN113407491A (en) Data processing method and device
CN105991450A (en) MAC address table updating method and device
EP3796598A1 (en) Node switching method, network node, network system, and storage medium
CN110855764A (en) Network traffic scheduling method and device and electronic equipment
CN112311833B (en) Data updating method and device
CN114125818B (en) Service parameter transmission method, device, system, electronic equipment and storage medium
CN113904953B (en) Offline detection method, device and equipment for communication equipment
CN109981660B (en) Information processing method and electronic equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant