CN110417898B - Data transmission method, device, client, electronic equipment and storage medium - Google Patents

Data transmission method, device, client, electronic equipment and storage medium Download PDF

Info

Publication number
CN110417898B
CN110417898B CN201910704055.3A CN201910704055A CN110417898B CN 110417898 B CN110417898 B CN 110417898B CN 201910704055 A CN201910704055 A CN 201910704055A CN 110417898 B CN110417898 B CN 110417898B
Authority
CN
China
Prior art keywords
data
message queue
transmission
messages
data storage
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910704055.3A
Other languages
Chinese (zh)
Other versions
CN110417898A (en
Inventor
周瑜
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Rajax Network Technology Co Ltd
Original Assignee
Rajax Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Rajax Network Technology Co Ltd filed Critical Rajax Network Technology Co Ltd
Priority to CN201910704055.3A priority Critical patent/CN110417898B/en
Publication of CN110417898A publication Critical patent/CN110417898A/en
Application granted granted Critical
Publication of CN110417898B publication Critical patent/CN110417898B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/182—Distributed file systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/14—Session management
    • H04L67/141—Setup of application sessions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/50—Network services
    • H04L67/56—Provisioning of proxy services
    • H04L67/566—Grouping or aggregating service requests, e.g. for unified processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L45/00—Routing or path finding of packets in data switching networks
    • H04L45/24—Multipath

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer And Data Communications (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The embodiment of the invention provides a data transmission method, a data transmission device, a client, electronic equipment and a storage medium, wherein the method comprises the following steps: collecting data from a data source; organizing the content of the data into messages in a target message queue; respectively establishing the target message queue and transmission channels of a plurality of specified data storage systems; and transmitting the messages of the target message queue to the plurality of data storage systems in parallel through the transmission channel. The embodiment of the invention can decouple the single transmission flow direction of the data, and realize the parallel expansion of the data transmission channels and the parallel transmission of the data in a plurality of transmission channels.

Description

Data transmission method, device, client, electronic equipment and storage medium
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a data transmission method, a data transmission device, a client, electronic equipment and a storage medium.
Background
Data transfer refers to collecting data from a data source and transferring the collected data to a DFS
A Distributed File System (Distributed File System) and the like. The transmission flow direction of the currently acquired data is single, that is, after the data is acquired from the data source, the data is transmitted to the data storage system through a single transmission channel. This data transmission may have the following subsequent problems:
data can only be read from a single data storage system, and the data reading way is limited; the disaster tolerance of the data is poor, and when a single data storage system has a problem, the data can be collected from the data source again.
Therefore, how to improve the data transmission manner, so as to provide a basis for solving the above problems, becomes a problem to be considered by those skilled in the art.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data transmission method, an apparatus, a client, an electronic device, and a storage medium, which achieve parallel transmission of data to multiple data storage systems through multiple transmission channels by decoupling a single transmission flow direction of the data, thereby providing a basis for solving the above mentioned problems.
In order to achieve the above purpose, the embodiments of the present invention provide the following technical solutions:
a method of data transmission, comprising:
collecting data from a data source;
organizing the content of the data into messages in a target message queue;
respectively establishing the target message queue and transmission channels of a plurality of specified data storage systems;
and transmitting the messages of the target message queue to the plurality of data storage systems in parallel through the transmission channel.
Optionally, the organizing the content of the data into the messages in the target message queue includes:
packaging the content of the data into event data according to rows;
and taking an event data as a message, and organizing the message in a target message queue in a queue form.
Optionally, the target message queue includes a first message queue corresponding to a first data transmission task; the data comprises first data corresponding to the first data transmission task.
Optionally, messages corresponding to different data transmission tasks in the target message queue are distinguished by different topics; the respectively establishing the transmission channels of the target message queue and the plurality of designated data storage systems comprises:
respectively establishing a target message queue and transmission channels of the appointed data storage systems according to different themes of the target message queue and aiming at different data transmission tasks;
the transmitting messages of the target message queue in parallel to the plurality of data storage systems through the transmission channel comprises:
and aiming at any data transmission task, transmitting the messages corresponding to the data transmission task in the target message queue to each data storage system in parallel through the transmission channel corresponding to the data transmission task.
Optionally, the target message queue further includes a second message queue corresponding to a second data transmission task; the data further comprises second data corresponding to the second data transmission task.
Optionally, the first message queue is different from the second message queue; the messages in the first message queue include: a first message of a content organization of the first data; the messages in the second message queue include: a second message of a content organization of the second data; and the first theme corresponding to the first message in the first message queue is different from the second theme corresponding to the second message in the second message queue.
An embodiment of the present invention further provides a data transmission device, including:
an acquisition module for acquiring data from a data source;
the message organization module is used for organizing the content of the data into messages in a target message queue;
the channel establishing module is used for respectively establishing the target message queue and the transmission channels of the plurality of specified data storage systems;
and the transmission module is used for transmitting the messages of the target message queue to the plurality of data storage systems in parallel through the transmission channel.
An embodiment of the present invention further provides a client, where the client is configured to execute the data transmission method described above.
An embodiment of the present invention further provides an electronic device, where the electronic device includes at least one memory and at least one processor, the memory stores a program, the processor invokes the program, and the program is configured to:
collecting data from a data source;
organizing the content of the data into messages in a target message queue;
respectively establishing the target message queue and transmission channels of a plurality of specified data storage systems;
and transmitting the messages of the target message queue to the plurality of data storage systems in parallel through the transmission channel.
An embodiment of the present invention further provides a storage medium, where the storage medium stores a program for executing the data transmission method.
The data transmission method provided by the embodiment of the invention can organize data collected from a data source into messages in a target message queue, so that the target message queue is used as a data transmission unit for data transmission, transmission channels of the target message queue and a plurality of designated data storage systems are respectively established in a data distribution mode, and then the messages of the target message queue are transmitted to the plurality of data storage systems in parallel through the established transmission channels, so that the data are transmitted to the plurality of data storage systems in parallel through the plurality of transmission channels, and the data are stored in the plurality of data storage systems in parallel.
According to the data transmission method provided by the embodiment of the invention, after data acquired by a data source is organized into messages in a target message queue, the target message queue is used as a data producer for providing the data, and a plurality of data storage systems are used as data consumers; data can be shunted from the target message queue in a data shunting manner, so that the data flows into a plurality of data storage systems, the single transmission flow direction of the data is decoupled, and the parallel expansion of data transmission channels and the parallel transmission of the data in a plurality of transmission channels are realized; furthermore, the embodiment of the invention can support a plurality of data storage systems to write data simultaneously, so that different data storage systems can store the whole amount of data, and a basis is provided for solving the problems of limited data reading path, poor data disaster tolerance and the like caused by single data transmission flow direction.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, it is obvious that the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a block diagram of a data transmission system according to an embodiment of the present invention;
fig. 2 is a flowchart of a data transmission method according to an embodiment of the present invention;
FIG. 3 is an exemplary diagram of data content organized as a message;
fig. 4 is an exemplary process diagram of a data transmission method according to an embodiment of the present invention;
fig. 5 is another block diagram of a data transmission system according to an embodiment of the present invention;
fig. 6 is another flowchart of a data transmission method according to an embodiment of the present invention;
fig. 7 is another exemplary process diagram of a data transmission method according to an embodiment of the present invention;
FIG. 8 is a diagram illustrating examples of data isolation protection provided by an embodiment of the present invention;
FIG. 9 is a flowchart of a data transmission method according to an embodiment of the present invention;
fig. 10 is a process diagram of another example of a data transmission method according to an embodiment of the present invention;
fig. 11 is a flowchart of a data transmission method according to an embodiment of the present invention;
fig. 12 is a process diagram of another example of a data transmission method according to an embodiment of the present invention;
fig. 13 is a block diagram of a data transmission apparatus according to an embodiment of the present invention;
fig. 14 is another block diagram of a data transmission apparatus according to an embodiment of the present invention;
fig. 15 is a further block diagram of a data transmission apparatus according to an embodiment of the present invention;
fig. 16 is a hardware block diagram of the electronic device.
Detailed Description
In the embodiment of the invention, in order to realize that data is transmitted to a plurality of data storage systems in parallel through a plurality of transmission channels, the content of the data collected from a data source can be organized into messages in MQ (Message Queue), so that the MQ is used as a data transmission unit, and the data is transmitted to the plurality of data storage systems in parallel through the plurality of transmission channels.
Based on this idea, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In an alternative implementation, fig. 1 shows an alternative block diagram of a data transmission system provided in an embodiment of the present invention, and as shown in fig. 1, the data transmission system may include: a client 10, a data source 20, a plurality of data storage systems 30, optionally, data storage systems such as DFS;
the client 10 is a client component on the data source side, and is used for collecting data from the data source; the client 10 may be a component implemented based on a data reading command, and data collection is implemented by executing the data reading command; in one example, the client 10 may be implemented based on taildir (a log view command) or kafka (a card, an open source streaming platform).
In the embodiment of the present invention, the client needs to implement the following functions in addition to data acquisition:
organizing the content of the collected data into messages in MQ; therefore, the client side takes the MQ as a data transmission unit, and transmits the messages in the MQ to a plurality of data storage systems in parallel through a plurality of transmission channels in a data distribution mode, so that the data can be transmitted to different data storage systems in parallel, and the data can be stored in parallel in multiple directions (namely the data can be stored in parallel in the plurality of data storage systems).
Optionally, the data transmission system shown in fig. 1 may be based on flume, which is a distributed, reliable and highly available system for collecting, aggregating and transmitting mass logs; of course, the data transmission system provided in the embodiment of the present invention may also be based on other distributed data acquisition, aggregation, and transmission systems, and is not limited to flash.
As an alternative implementation, based on the data acquisition system shown in fig. 1, standing at the perspective of the client, fig. 2 shows an alternative flow of the data transmission method provided in the embodiment of the present invention, and as shown in fig. 2, the flow may include:
and step S10, collecting data from the data source.
The embodiment of the invention does not limit the way in which the client acquires data from the data source.
Step S11, organizing the content of the data as a message in the target MQ.
Alternatively, embodiments of the invention may pre-configure a plurality of MQs, the target MQ may be at least one of the plurality of MQs; in an alternative implementation, embodiments of the present invention may specify a target MQ, such that a client may select the specified target MQ from a plurality of MQs that are preconfigured; in other possible implementations, the target MQ may not be specified, for example, an MQ that can accommodate the data may be selected from a preconfigured plurality of MQs as the target MQ.
It is to be appreciated that MQ is comprised of messages organized in queues, upon which embodiments of the present invention can convert the content of the collected data into messages in MQ and organize the messages in queues within MQ; in an optional implementation of step S11, the embodiment of the present invention may encapsulate the content of the data collected from the data source into event data by rows, where one event data may correspond to one message, so that organizing the content of the data into messages in the target MQ is implemented by organizing the one event data as one message in the target MQ in the form of a queue;
for example, as shown in fig. 3, assuming that data a is data that a client needs to collect from a data source, the client may package the content of the data a into event data by rows, that is, one row of the content of the data a is packaged into one event data, and the packaged one event data may serve as one message, so that as the client continuously packages the content of the data a into event data by rows, the message of the event data may be organized in a queue form in a target MQ.
Of course, the example shown in fig. 3 is only one alternative way for the embodiment of the present invention to organize the content of the data as a message in the target MQ, and the embodiment of the present invention is not limited to other possible ways as long as the processed data content can serve as a message supported by the target MQ and the purpose of organizing the message in the form of a queue is achieved.
After the content of the data is organized into the message in the target MQ, the target MQ can be used as a new data producer (producer) in the embodiment of the invention, namely, the target MQ is used as a new data source in the embodiment of the invention, and the target MQ provides the data to the data storage system; because the essence of the target MQ providing data to the data storage system is that the messages in the target MQ are provided to the data storage system and the messages in the target MQ are organized in the form of queues, the parallel transmission of the messages in the target MQ is possible; based on this, the embodiment of the present invention may transmit the message in the target MQ to multiple data storage systems in parallel through multiple transmission channels in a data offloading manner, and an alternative implementation may be as shown in step S12 and step S13 below with reference to fig. 2.
And step S12, respectively establishing transmission channels of the target MQ and the plurality of specified data storage systems.
The embodiment of the invention can appoint a plurality of data storage systems for receiving data, so that after organizing the content of the collected data into the message in the target MQ, the client can respectively establish the transmission channels of the target MQ and the appointed data storage systems; for example, if the number of the target MQs is multiple, the transmission channels between each target MQ and each data storage system need to be established respectively; if the number of the target MQ is two, including MQ1 and MQ2, it is necessary to establish a transmission channel with each data storage system for MQ1 and a transmission channel with each data storage system for MQ 2.
And step S13, transmitting the messages of the target MQ to the plurality of data storage systems in parallel through the established transmission channels.
After the transmission channels of the target MQ and the data storage systems are established, message transmission in the target MQ can be carried out, and the message in the target MQ is transmitted to the data storage systems in parallel through the established transmission channels, so that the message in the target MQ can be stored in the data storage systems in parallel, namely the data collected by the full amount of clients can be stored in the data storage systems.
For convenience of illustration, the data transmission method provided by the embodiment of the present invention is illustrated below by taking the message in the target MQ as event data, the data storage system as DFS, and the DFS1 and DFS2 as examples for receiving data;
with reference to fig. 3 and 4, after encapsulating the content of data a into event data by rows and organizing one event data as a message in a queue, if the event data flows out of the target MQ, the target MQ becomes the data producer of the event data, i.e., the target MQ becomes the new data source of the embodiment of the present invention, and DFS1 and DFS2 can be regarded as the data consumer of the target MQ, i.e., DFS1 and DFS2 consume the event data in the target MQ;
thus, embodiments of the present invention may establish transport channel 1 of the target MQ and DFS1 and transport channel 2 of the target MQ and DFS2, and transmit the message in the target MQ to DFS1 through transport channel 1 and to DFS2 through transport channel 2, implementing parallel transmission of the message in the target MQ over multiple transport channels, so that DFS1 and DFS2 may store the full amount of data a.
It can be seen that the data transmission method provided in the embodiment of the present invention can organize data collected from a data source into messages in a target MQ, so as to perform data transmission with the target MQ as a data transmission unit, respectively establish transmission channels between the target MQ and a plurality of specified data storage systems in a data splitting manner, and further transmit the messages of the target MQ in parallel to the plurality of data storage systems through each established transmission channel, thereby implementing parallel transmission of the data to the plurality of data storage systems through the plurality of transmission channels, and enabling the data to be stored in parallel in the plurality of data storage systems.
The data transmission method provided by the embodiment of the invention organizes the data collected by the data source into the message in the target MQ, and then takes the target MQ as a data producer providing the data and a plurality of data storage systems as data consumers; data can be shunted from the target MQ in a data shunting manner, so that the data flows into a plurality of data storage systems, the single transmission flow direction of the data is decoupled, and the parallel expansion of data transmission channels and the parallel transmission of the data in a plurality of transmission channels are realized; furthermore, the embodiment of the invention can support a plurality of data storage systems to write data simultaneously, so that different data storage systems can store the whole amount of data, and a basis is provided for solving the problems of limited data reading path, poor data disaster tolerance and the like caused by single data transmission flow direction.
Specifically, with reference to fig. 4, by using the data transmission method provided by the embodiment of the present invention, DFS1 and DFS2 can store the entire amount of data, so that when a user reads data, the user can read data from DFS1 and DFS2, thereby implementing the extension of data reading path; in addition, when problems occur in the DFS1, because the DFS2 stores full data, data disaster tolerance can be realized through the DFS2 without re-collecting data from a data source; of course, in practical applications, the number of the data storage systems may be plural, and is not limited to the two DFSs shown in fig. 4.
Further, with the data transmission method provided by the embodiment of the present invention, smooth transition of multiple data storage systems can be realized, for example, in the process of transition from DFS1 to DFS2, with the data transmission method provided by the present invention, since DFS1 and DFS2 hold full data, smooth transition from DFS1 to DFS2 can be realized, and after the transition is completed, DFS1 is taken off line without worrying about data loss.
Therefore, the embodiment of the invention can improve the flexibility of the data service on the basis of decoupling the single transmission flow direction of the data, realizing the parallel expansion of the data transmission channels and the parallel transmission of the data in a plurality of transmission channels, so that the data service has more possibilities.
In an optional implementation, the client may implement a data transmission process based on task information sent by the server, and optionally, fig. 5 shows another optional block diagram of the data transmission system provided in the embodiment of the present invention, and as shown in fig. 1 and fig. 5, the data transmission system may further include: a server 40; the server 40 is a service component on the user side, and when the user has a data transmission requirement, the user can configure task information of a data transmission task at the server 40, and after the server 40 sends the task information to the client 10, the client can execute the flow shown in fig. 2, and complete the data transmission task in a manner of parallel transmission of data in multiple transmission channels.
On one hand, based on the data transmission system shown in fig. 5, the data transmission method provided by the embodiment of the invention can be used for executing any data transmission task; taking the execution of the first data transmission task as an example, the target MQ in the flow shown in fig. 2 may be a first MQ corresponding to the first data transmission task, and the data collected from the data source may be first data corresponding to the first data transmission task; optionally, standing at the perspective of the client, fig. 6 shows another optional flow of the data transmission method provided in the embodiment of the present invention, and as shown in fig. 6, the flow may include:
and step S20, acquiring first task information corresponding to the first data transmission task.
Optionally, the first data transmission task may be any data transmission task, and in the embodiment of the present invention, the first task information may be configured by the user at the server, and the server sends the first task information to the client.
In one example, the first task information may indicate a first data source, which needs to collect data, corresponding to the first data transmission task, and first data which needs to be collected from the first data source; for example, the first task information may include a first data source identifier and a first data identifier, wherein the first data source identifier may be used to indicate a first data source, the first data identifier may be used to indicate first data that needs to be collected from the first data source, and the data identifier may be information of identifiable data such as a data type.
And step S21, collecting first data from a first data source according to the first task information.
After the client acquires the first task information, the client can acquire data from a first data source according to a first data source indicated by the first task information and first data needing to be acquired from the first data source; for example, the client may link a first data source corresponding to the first data source identifier according to the first data source identifier carried in the first task information, and read first data corresponding to the first data identifier from the first data source according to the first data identifier carried in the first task information, so as to acquire first data corresponding to a first data transmission task.
It should be noted that, steps S20 and S21 are only an alternative way for the client to collect data from the data source, and the client may collect data from the data source periodically or collect real-time data from the data source after a task is configured.
Step S22, organizing the content of the first data as a first message in a first MQ.
The first MQ is an optional form of the target MQ, and in an optional implementation, embodiments of the invention may specify a first MQ corresponding to the first data transfer task, e.g., carrying a first MQ identification in the first task information, so that the client may select the first MQ corresponding to the first MQ identification from pre-configuring the plurality of MQs; it should be noted that the number of the first MQ may be multiple, for example, the multiple first MQ identifiers correspond to the multiple first MQ identifiers;
in another alternative implementation, the first MQ corresponding to the first data transfer task may also not be indicated by the first task information, e.g., the client may select an MQ from a plurality of MQs that are preconfigured to accommodate the first data as the first MQ when performing a single data transfer task.
After the client selects the first MQ, step S22 may be executed to organize the content of the first data as a first message in the first MQ; optionally, the manner of organizing the content of the first data into the first message in the first MQ may refer to the foregoing description, such as encapsulating the content of the first data into event data by rows, so as to treat one event data as one first message, and organizing the first message in the form of a queue in the first MQ; it should be noted that the first message referred to herein is a message corresponding to event data into which the content of the first data is encapsulated, and the number of the first messages may be multiple.
And step S23, respectively establishing transmission channels of the first MQ and the plurality of appointed data storage systems.
Optionally, the specified multiple data storage systems may also be indicated by the first task information, for example, the first task information may carry multiple data storage system identifiers, and one data storage system identifier corresponds to one data storage system; of course, the plurality of data storage systems may also be a default data storage system designated in advance without being instructed by the first task information.
And step S24, transmitting the first message in the first MQ to the plurality of data storage systems in parallel through the established transmission channels.
In one example, illustrated with the description of the first task information of the first data transfer task as "transfer transaction log data for app1 to DFS1 and DFS2 using MQ 1", then the database whose data source is app1, the collected data is transaction log data, the target MQ used is MQ1, and the data storage systems storing the data are DFS1 and DFS 2; based on the example first task information, after the client receives the first task information, fig. 7 shows another optional example process of the data transmission method provided by the embodiment of the present invention, and as shown in fig. 7, the example process may include:
step S30, client links the database of app 1.
Step S31, the client reads the transaction log data from the database of app 1.
Step S32, the client selects MQ1 from the preconfigured MQs 1-MQn, organizes the contents of the transaction log data as a first message in MQ 1.
Step S33, the client establishes a transmission channel between MQ1 and DFS1, and transmits the first message in MQ1 to DFS1 through the established transmission channel of DFS 1.
Step S34, the client establishes a transmission channel between MQ1 and DFS2, and transmits the first message in MQ1 to DFS2 through the established transmission channel of DFS 2.
Alternatively, step S33 and step S34 may be performed synchronously.
It can be seen that in an optional implementation of the embodiment of the present invention, the client may determine, based on the indication of the first task information, the target MQ and the plurality of data storage systems of the data transmission destination, so that after organizing the data collected by the data source into the messages in the target MQ, the client may perform data offloading from the target MQ, so that the data flows into the plurality of data storage systems indicated by the task information, thereby decoupling the single transmission flow direction of the data, and implementing parallel extension of the data transmission channels and parallel transmission of the data in the plurality of transmission channels.
In an optional implementation, the data transmission method provided in the embodiment of the present invention may be used to simultaneously execute a plurality of data transmission tasks; when a plurality of data transmission tasks are executed simultaneously, the embodiment of the invention ensures that transmission channels for transmitting data of different data transmission tasks are not overlapped so as to realize the isolation protection of the data of different data transmission tasks during transmission;
as an optional implementation, messages corresponding to different data transmission tasks in the target MQ may be distinguished by different topics (subjects), that is, messages of different topics in the target MQ correspond to different data transmission tasks, so that embodiments of the present invention may respectively establish transmission channels of the target MQ and each specified data storage system for different data transmission tasks according to different topics of the target MQ, so that each data transmission task has a respective transmission channel, that is, each data transmission task has a plurality of transmission channels from the target MQ to each specified data storage system, and the transmission channel corresponding to each data transmission task is identified by the respective topc in the target MQ;
therefore, for any data transmission task, the embodiment of the present invention may transmit the message corresponding to the data transmission task in the target MQ to each data storage system in parallel through the transmission channel corresponding to the data transmission task, so that the data of each data transmission task is transmitted to each data storage system in parallel on the respective transmission channel, and further, on the basis that the data of each data transmission task is transmitted in parallel through multiple transmission channels, the isolation protection of the data of different data transmission tasks is realized.
For ease of understanding, fig. 8 illustrates an exemplary data isolation protection manner, and as shown in fig. 8, topic of message 1 corresponding to data transfer task 1 in the target MQ is topic1, topc of message 2 corresponding to data transfer task 2 is topc 2, i.e., message 1 corresponding to data transfer task 1 and message 2 corresponding to data transfer task 2 in the target MQ are distinguished by topc 1 and topc 2; in the embodiment of the present invention, the number of the target MQ may be at least one, message 1 of topic1 and message 2 of topic2 may exist in the same MQ or may exist in different MQs, and the number of message 1 and message 2 may be multiple;
therefore, the embodiment of the invention can establish the target MQ and the transmission channel 1 of each data storage system according to the topc 1, and obtain the target MQ corresponding to the data transmission task 1 to the plurality of transmission channels 1 of each data storage system; establishing a target MQ and a transmission channel 2 of each data storage system according to the topic2 to obtain a plurality of transmission channels 2 from the target MQ corresponding to the data transmission task 2 to each data storage system; because the transmission channel 1 corresponding to the data transmission task 1 is established based on the topic1 and can be identified by the topic1, the message 1 of the topc 1 can be transmitted in the transmission channel 1, and the message 2 of the topc 2 is forbidden to be transmitted in the transmission channel 1, so that when the message 1 corresponding to the data transmission task 1 is transmitted to each data storage system in parallel through a plurality of transmission channels 1, the data isolation protection effect during the transmission of the message 1 is achieved;
similarly, since the transmission channel 2 corresponding to the data transmission task 2 is established based on the topic2 and can be identified by the topic2, the message 2 of the topc 2 can be transmitted on the transmission channel 2, and the message 1 of the topc 1 can be prohibited from being transmitted on the transmission channel 2.
The embodiment of the invention can provide different data isolation protection schemes aiming at whether the messages of different topics exist in the same MQ. As an alternative implementation, to simultaneously perform multiple data transfer tasks includes: for example, if the first data transmission task and the second data transmission task are executed, the target MQ in the process shown in fig. 2 may be a first MQ corresponding to the first data transmission task and a second MQ corresponding to the second data transmission task, and the data collected from the data source may be first data corresponding to the first data transmission task and second data corresponding to the second data transmission task; furthermore, the embodiment of the present invention may provide different data isolation protection schemes for determining whether the first MQ and the second MQ are the same, that is, for determining whether the same MQ is used to transmit data of different data transmission tasks, the embodiment of the present invention may provide different data isolation protection schemes.
In an alternative implementation, the first MQ and the second MQ may be the same MQ, e.g., both first MQs, i.e., embodiments of the present invention may support the transfer of data for different data transfer tasks using the same MQ; optionally, from the perspective of a client, fig. 9 shows another optional flow of the data transmission method provided in the embodiment of the present invention, and as shown in fig. 9, the flow may include:
and step S40, acquiring the first task information and the second task information.
The first task information may be task information of the first data transmission task, and the second task information may be task information of the second data transmission task. The first task information and the second task information can be sent to the client by the server after the user configures the server.
And step S41, acquiring first data from a first data source according to the first task information, and acquiring second data from a second data source according to the second task information.
Optionally, the first task information may include a first data source identifier and a first data identifier, and the second task information may include a second data source identifier and a second data identifier; the client can link a first data source corresponding to the first data source identification according to the first data source identification, and read first data corresponding to the first data identification from the first data source according to the first data identification; meanwhile, the client may link a second data source corresponding to the second data source identifier according to the second data source identifier, and read second data corresponding to the second data identifier from the second data source according to the second data identifier.
Optionally, in this embodiment of the present invention, the data source for reading the first data and the second data may be the same, for example, the first data source is the same as the second data source, and accordingly, the client may collect different first data and second data from the same data source. Of course, the first data source may be different from the second data source.
Step S42, organize the content of the first data as a first message in the first MQ, organize the content of the second data as a second message in the first MQ, and set a first topic corresponding to the first message in the first MQ, which is different from a second topic corresponding to the second message.
Since the first MQ is identical to the second MQ, the description of the embodiments of the present invention is made using the first MQ; optionally, the first MQ may be indicated by the task information, that is, both the first task information and the second task information may indicate the first MQ, for example, both the first task information and the second task information may carry the first MQ identifier. Of course, the first MQ may not be indicated by the task information, for example, in the case where the default setting uses the same MQ to transfer the data of the first data transfer task and the second data transfer task, embodiments of the present invention may select the first MQ having the data capacity capable of accommodating the first data and the second data from a plurality of MQs configured in advance.
The embodiment of the invention can organize the content of the first data into the first message in the first MQ, set the first topic corresponding to the first message in the first MQ, organize the content of the second data into the second message in the first MQ, and set the second topic corresponding to the second message in the first MQ; since the first message is a message corresponding to the first data in the first MQ, the second message is a message corresponding to the second data in the first MQ, and the first data and the second data belong to different data transmission tasks, that is, the first message corresponds to the first data transmission task, and the second message corresponds to the second data transmission task, the embodiment of the present invention may set the first topic to be different from the second topic, so as to distinguish the first message from the second message in the first MQ;
it should be noted that the first message does not refer to one message, but refers to a message of content organization of the first data, and the number of the first messages may be multiple; similarly, the second message refers to a message in which the content of the second data is organized, and the number of the second messages may be plural.
Optionally, for the first data, the embodiment of the present invention may encapsulate the content of the first data into event data by rows, so as to regard one event data as one first message, and organize the first message in the form of a queue in the first MQ; for the second data, the embodiment of the present invention may encapsulate the content of the second data into event data by rows, so that one event data is used as one second message, and the second message is organized in a queue form in the first MQ; the manner in which the content of the data is organized into messages is the same as that described above and can be referred to one another.
Step S43, establishing first transmission channels between the first MQ and the plurality of data storage systems respectively according to the first topic, and establishing second transmission channels between the first MQ and the plurality of data storage systems respectively according to the second topic.
Because the first data and the second data belong to different data transmission tasks and need to be transmitted in an isolated manner, under the condition that a first topic of a first message in the first MQ is set to be different from a second topic of a second message, the embodiment of the invention can carry out transmission isolation protection on the data of different tasks through the topic; specifically, the client may establish, according to the first topic, first transmission channels between the first MQ and the multiple data storage systems, respectively, so that the first transmission channels may be identified by the first topic and used to transmit a first message corresponding to the first data transmission task, and the second message having the second topic cannot be transmitted in the first transmission channel identified by the first topic;
meanwhile, the client may establish, according to the second topic, second transmission channels between the first MQ and the plurality of data storage systems, respectively, so that the second transmission channels may be identified by the second topic and used for transmitting a second message corresponding to the second data transmission task, and the first message having the first topic cannot be transmitted in the second transmission channel identified by the second topic.
It can be understood that the first transmission channel is a transmission channel corresponding to the first data transmission task, and the second transmission channel is a transmission channel corresponding to the second data transmission task. The first transmission channel does not refer to one transmission channel, but based on the first topic, the first MQ and each data storage system establish the first transmission channel, and the number of the first transmission channels can be multiple; similarly, based on the second topic, the first MQ and each data storage system establish a second transmission channel, and the number of the second transmission channels may be multiple.
Step S44, concurrently transmitting the first message of the first MQ to the plurality of data storage systems through the established first transmission channel, and concurrently transmitting the second message of the first MQ to the plurality of data storage systems through the established second transmission channel.
The first message of the first MQ is transmitted in parallel by using the first transmission channel, the first message can be transmitted to the plurality of data storage systems in parallel, and the parallel transmission of the first data transmission task through the plurality of transmission channels is realized, so that the first data are stored in the plurality of data storage systems in parallel; and the second message of the first MQ is transmitted in parallel by using the second transmission channel, the second message can be transmitted to the plurality of data storage systems in parallel, and the parallel transmission of the second data transmission task through the plurality of transmission channels is realized, so that the second data are stored in the plurality of data storage systems in parallel.
Therefore, the embodiment of the invention can decouple the single transmission flow direction of the data under the condition of distinguishing the data of different tasks, so that the data of different tasks can be transmitted to a plurality of data storage systems in parallel through a plurality of transmission channels under the condition of isolation, and the data transmission quality is improved.
As an example of performing multiple data transfer tasks using the same MQ, with the description of the first task information of the first data transfer task as "transferring transaction log data of app1 to DFS1 and DFS2 using MQ 1", and the description of the second task information of the second data transfer task as "browsing data of a user of app1 to DFS1 and DFS2 using MQ 1", for illustration, fig. 10 shows yet another optional example procedure of a data transfer method provided by an embodiment of the present invention, which may include, as shown in fig. 10:
step S50, client links the database of app 1.
In step S51, the client reads the transaction log data and the user browsing data from the database of app 1.
Alternatively, the transaction log data may be an alternative form of the first data and the user browsing data may be an alternative form of the second data.
Step S52, the client selects MQ1 from the pre-configured MQ1-MQn, organizes the content of the transaction log data as a first message in MQ1, and sets a first topic of the first message.
Step S53, the client organizes the content of the user browsing data as a second message in the MQ1, and sets a second topic of the second message, which is different from the first topic.
Alternatively, step S52 and step S53 may be performed synchronously.
Step S54, the client establishes a first transmission channel between MQ1 and DFS1 according to the first topic, and transmits the first message in MQ1 to DFS1 through the first transmission channel.
Step S55, the client establishes a first transmission channel between MQ1 and DFS2 according to the first topic, and transmits the first message in MQ1 to DFS2 through the first transmission channel.
Alternatively, step S54 and step S55 may be performed synchronously.
And step S56, the client establishes a second transmission channel between the MQ1 and the DFS1 according to the second topic, and transmits the second message in the MQ1 to the DFS1 through the second transmission channel.
And step S57, the client establishes a second transmission channel between the MQ1 and the DFS2 according to the second topic, and transmits the second message in the MQ1 to the DFS2 through the second transmission channel.
Alternatively, step S56 and step S57 may be performed synchronously; steps S54 through S57 may also be performed simultaneously.
Therefore, when the same MQ is used for transmitting data of different data transmission tasks, the embodiment of the invention can distinguish the data of different tasks through different topics in the MQ, thereby establishing different transmission channels for the different tasks based on the topics, realizing the data transmission isolation of the different tasks and ensuring the quality of data transmission.
Alternatively, the first MQ and the second MQ may be different MQs, i.e., embodiments of the present invention may use different MQs to transfer data for different data transfer tasks; from the perspective of the client, fig. 11 shows another optional flow of the data transmission method provided by the embodiment of the present invention, and as shown in fig. 11, the flow may include:
and step S60, acquiring the first task information and the second task information.
And step S61, acquiring first data from a first data source according to the first task information, and acquiring second data from a second data source according to the second task information.
Alternatively, the description of step S60 can be referred to in the previous step S40, and the description of step S61 can be referred to in the previous step S41, which will not be described again.
Step S62, organize the content of the first data as a first message in the first MQ, organize the content of the second data as a second message in the second MQ, and set a first topic corresponding to the first message in the first MQ, which is different from a second topic corresponding to the second message in the second MQ.
Optionally, the first MQ may be indicated by the first task information, for example, the first task information may carry a first MQ identity, so that the client may select, from the preconfigured plurality of MQs, the first MQ corresponding to the first MQ identity; the second MQ may be indicated by the second task information, for example, the second task information may carry a second MQ identifier, so that the client may select, from the preconfigured plurality of MQs, the second MQ corresponding to the second MQ identifier;
of course, the first MQ and the second MQ may not be indicated by the task information, for example, in the case where the default setting uses different MQs to transfer the data of the first data transfer task and the second data transfer task, embodiments of the present invention may select the first MQ whose data capacity can accommodate the first data and the second MQ whose data capacity can accommodate the second data from among the preconfigured plurality of MQs.
Optionally, for the first data, the embodiment of the present invention may encapsulate the content of the first data into event data by rows, so as to regard one event data as one first message, and organize the first message in the form of a queue in the first MQ; for the second data, the embodiment of the present invention may encapsulate the content of the second data into event data by rows, so as to regard one event data as one second message, and organize the second message in the form of a queue in the second MQ; the manner in which the content of the data is organized into messages is the same as that described above and can be referred to one another.
Since the first message corresponds to the first data of the first data transmission task and the second message corresponds to the second data of the second data transmission task, the embodiment of the present invention may set the first topic of the first message in the first MQ to be different from the second topic of the second message in the second MQ, so as to distinguish the first message and the second message of different tasks.
Step S63, establishing first transmission channels between the first MQ and the plurality of data storage systems respectively according to the first topic, and establishing second transmission channels between the second MQ and the plurality of data storage systems respectively according to the second topic.
Because the first data corresponding to the first data transmission task and the second data corresponding to the second data transmission task need to be transmitted in an isolation mode, under the condition that the first topic of the first message in the first MQ is set to be different from the second topic of the second message in the second MQ, the embodiment of the invention can carry out transmission isolation protection on the data of different tasks through the topic; specifically, the client may establish first transmission channels between the first MQ and the plurality of data storage systems according to the first topic, so that the first transmission channels may be identified by the first topic and used for transmitting the first message in the first MQ, so that the second message in the second MQ having the second topic cannot be transmitted in the first transmission channel identified by the first topic;
meanwhile, the client may establish second transmission channels between the second MQ and the plurality of data storage systems respectively according to the second topic, so that the second transmission channels may be identified by the second topic for transmitting the second messages in the second MQ, so that the first messages having the first topic in the first MQ cannot be transmitted in the second transmission channels identified by the second topic.
It is to be understood that the first transmission channel may be considered as a transmission channel corresponding to the first data transmission task, and the second transmission channel may be considered as a transmission channel corresponding to the second data transmission task.
Step S64, concurrently transmitting the first message of the first MQ to the plurality of data storage systems through the established first transmission channel, and concurrently transmitting the second message of the second MQ to the plurality of data storage systems through the established second transmission channel.
Transmitting the first message of the first MQ in parallel using the first transmission channel, the first message being transmitted to the plurality of data storage systems in parallel such that the first data is stored in parallel in the plurality of data storage systems; the second message of the second MQ is transmitted in parallel using the second transmission channel, such that the second data is stored in parallel in the plurality of data storage systems.
As an example of performing multiple data transfer tasks using different MQs, with the description of the first task information as "transferring transaction log data of app1 to DFS1 and DFS2 using MQ 1", and the description of the second task information as "browsing data of a user of app1 to DFS1 and DFS2 using MQ 2", for illustration, fig. 12 shows yet another alternative example procedure of the data transfer method provided by an embodiment of the present invention, which may include, as shown in fig. 12:
step S70, client links the database of app 1.
In step S71, the client reads the transaction log data and the user browsing data from the database of app 1.
Step S72, the client selects MQ1 from the pre-configured MQs 1-MQn, organizes the content of the transaction log data into a first message in MQ1, and sets a first topic of the first message.
Step S73, the client selects MQ2, organizes the content of the user browsing data into a second message in MQ2, and sets a second topic of the second message, wherein the first topic is different from the second topic.
Alternatively, step S72 and step S73 may be performed synchronously.
Step S74, the client establishes a first transmission channel between MQ1 and DFS1 according to the first topic, and transmits the first message in MQ1 to DFS1 through the first transmission channel.
Step S75, the client establishes a first transmission channel between MQ1 and DFS2 according to the first topic, and transmits the first message in MQ1 to DFS2 through the first transmission channel.
Alternatively, step S74 and step S75 may be performed synchronously.
And step S76, the client establishes a second transmission channel between the MQ2 and the DFS1 according to the second topic, and transmits the second message in the MQ2 to the DFS1 through the second transmission channel.
And step S77, the client establishes a second transmission channel between the MQ2 and the DFS2 according to the second topic, and transmits the second message in the MQ2 to the DFS2 through the second transmission channel.
Alternatively, step S76 and step S77 may be performed synchronously. Steps S74-S77 may also be performed synchronously.
The data transmission method provided by the embodiment of the invention can decouple the single transmission flow direction of the data, realize the parallel expansion of the data transmission channels and the parallel transmission of the data in a plurality of transmission channels, improve the flexibility of the data service and enable the data service to have more possibilities. Further, the data transmission method provided by the embodiment of the present invention can distinguish data of different tasks through the topic of the MQ, so that different transmission channels are established for different tasks based on the topic, data transmission isolation of different tasks is realized, and the quality of data transmission is ensured.
While various embodiments of the present invention have been described above, various alternatives described in the various embodiments can be combined and cross-referenced without conflict to extend the variety of possible embodiments that can be considered disclosed and disclosed in connection with the embodiments of the present invention.
In the following, a data transmission device provided in the embodiment of the present invention is described, and the data transmission device described below may be considered as a functional module that is required to implement the data transmission method provided in the embodiment of the present invention. The contents of the data transmission device described below may be referred to in correspondence with the contents described above.
Fig. 13 is a block diagram of a data transmission device according to an embodiment of the present invention, and as shown in fig. 13, the data transmission device may include:
an acquisition module 100 for acquiring data from a data source;
a message organization module 110, configured to organize the content of the data into messages in a target message queue;
a channel establishing module 120, configured to respectively establish the target message queue and transmission channels of the multiple specified data storage systems;
and a transmission module 130, configured to transmit the messages of the target message queue to the multiple data storage systems in parallel through the transmission channel.
Optionally, the message organizing module 110 is configured to organize the content of the data into a message in a target message queue, and may specifically include:
packaging the content of the data into event data according to rows;
and taking an event data as a message, and organizing the message in a target message queue in a queue form.
Optionally, when a single data transmission task or multiple data transmission tasks are executed, taking the executed data transmission task as an example including a first data transmission task, the target message queue may include a first message queue, and the first message queue may be a message queue corresponding to the first data transmission task; the data may include first data corresponding to the first data transfer task.
Optionally, fig. 14 shows another block diagram of the data transmission device according to the embodiment of the present invention, and in combination with fig. 13 and 14, the data transmission device may further include:
a task obtaining module 140, configured to obtain first task information corresponding to the first data transmission task, where the first task information at least includes a first data source identifier and a first data identifier;
optionally, the acquisition module 100 is configured to acquire data from a data source, and may specifically include:
linking a first data source corresponding to the first data source identifier according to the first data source identifier;
and reading first data corresponding to the first data identifier from the first data source according to the first data identifier.
Optionally, further, the first task information may further include: a first message queue identity; based on this, fig. 15 shows still another block diagram of the data transmission device according to the embodiment of the present invention, and as shown in fig. 14 and fig. 15, the data transmission device may further include:
a selecting module 150, configured to select, from a plurality of preconfigured message queues, a first message queue corresponding to the first message queue identifier before the message organizing module 110 organizes the content of the data as a message in a target message queue.
Optionally, when a plurality of data transmission tasks are executed, messages corresponding to different data transmission tasks in the target message queue are distinguished by different topics; optionally, the channel establishing module 120 is configured to respectively establish the target message queue and transmission channels of the multiple specified data storage systems, and may specifically include:
respectively establishing a target message queue and transmission channels of the appointed data storage systems according to different themes of the target message queue and aiming at different data transmission tasks;
the transmission module 130 is configured to transmit the messages in the target message queue to the multiple data storage systems in parallel through the transmission channel, and may specifically include:
and aiming at any data transmission task, transmitting the messages corresponding to the data transmission task in the target message queue to each data storage system in parallel through the transmission channel corresponding to the data transmission task.
Optionally, taking an example that the executed data transmission task includes a first data transmission task and a second data transmission task, the target message queue may further include a second message queue, and the second message queue may be a message queue corresponding to the second data transmission task; the collected data may further include second data corresponding to the second data transmission task.
In one aspect, embodiments of the present invention support the use of the same message queue to transmit data of different data transmission tasks, where, for example, the first message queue is the same as the second message queue, and based on this, the messages in the first message queue include: a first message of content organization of the first data, and a second message of content organization of the second data; and the first theme corresponding to the first message in the first message queue is different from the second theme corresponding to the second message.
Optionally, the channel establishing module 120 is configured to respectively establish the target message queue and transmission channels of the specified data storage systems for different data transmission tasks according to different topics of the target message queue, and may specifically include:
establishing a first message queue and first transmission channels of a plurality of data storage systems respectively according to the first theme, and establishing a first message queue and second transmission channels of the plurality of data storage systems respectively according to the second theme;
the transmission module 130 is configured to, for any data transmission task, transmit a message corresponding to the data transmission task in the target message queue to each data storage system in parallel through a transmission channel corresponding to the data transmission task, and may specifically include:
and transmitting the first messages of the first message queue to the plurality of data storage systems in parallel through the established first transmission channel, and transmitting the second messages of the first message queue to the plurality of data storage systems in parallel through the established second transmission channel.
Optionally, in another aspect, in the embodiment of the present invention, it may be supported to use different message queues to transmit data of different data transmission tasks, for example, the first message queue is different from the second message queue, based on which, the messages in the first message queue include: a first message of a content organization of the first data; the messages in the second message queue include: a second message of a content organization of the second data; and the first theme corresponding to the first message in the first message queue is different from the second theme corresponding to the second message in the second message queue.
Optionally, the channel establishing module 120 is configured to respectively establish the target message queue and transmission channels of the specified data storage systems for different data transmission tasks according to different topics of the target message queue, and may specifically include:
respectively establishing a first message queue and first transmission channels of a plurality of data storage systems according to the first theme, and respectively establishing a second message queue and second transmission channels of the plurality of data storage systems according to the second theme;
the transmission module 130 is configured to, for any data transmission task, transmit a message corresponding to the data transmission task in the target message queue to each data storage system in parallel through a transmission channel corresponding to the data transmission task, and may specifically include:
and transmitting the first messages of the first message queue to the plurality of data storage systems in parallel through the established first transmission channel, and transmitting the second messages of the second message queue to the plurality of data storage systems in parallel through the established second transmission channel.
The data transmission device provided by the embodiment of the invention can be applied to a client, and the client can execute the data transmission method provided by the embodiment of the invention by setting the data transmission device in a program form.
The embodiment of the invention also provides the electronic equipment, which can be the electronic equipment where the client is located; alternatively, the hardware structure of the electronic device may be as shown in fig. 16, and includes: at least one processor 01, at least one communication interface 02, at least one memory 03 and at least one communication bus 04;
in the embodiment of the present invention, the processor 01, the communication interface 02, and the memory 03 complete mutual communication through the communication bus 04;
processor 01 may be a central processing unit CPU, or an Application Specific Integrated Circuit (ASIC), or one or more Integrated circuits configured to implement an embodiment of the invention.
The memory 03 may comprise a high-speed RAM memory and may further comprise a non-volatile memory, such as at least one disk memory.
The memory 03 stores a program, and the processor 01 calls the program stored in the memory 3 to execute the data transmission method provided by the embodiment of the present invention.
The embodiment of the invention also provides a storage medium, and the storage medium can store a program for executing the data transmission method provided by the embodiment of the invention.
Optionally, the program may be specifically configured to:
collecting data from a data source;
organizing the content of the data into messages in a target message queue;
respectively establishing the target message queue and transmission channels of a plurality of specified data storage systems;
and transmitting the messages of the target message queue to the plurality of data storage systems in parallel through the transmission channel.
The detailed function and the extended function of the program can be referred to the description of the corresponding parts.
Although the embodiments of the present invention have been disclosed, the present invention is not limited thereto. Various changes and modifications may be effected therein by one skilled in the art without departing from the spirit and scope of the invention as defined in the appended claims.
In summary, in a first aspect, an embodiment of the present invention provides a data transmission method, including:
collecting data from a data source;
organizing the content of the data into messages in a target message queue;
respectively establishing the target message queue and transmission channels of a plurality of specified data storage systems;
and transmitting the messages of the target message queue to the plurality of data storage systems in parallel through the transmission channel.
In a second aspect, according to the data transmission method in the first aspect, the organizing the content of the data into messages in a target message queue includes:
packaging the content of the data into event data according to rows;
and taking an event data as a message, and organizing the message in a target message queue in a queue form.
In a third aspect, according to the data transmission method in the first or second aspect, the target message queue includes a first message queue corresponding to a first data transmission task; the data comprises first data corresponding to the first data transmission task.
In a fourth aspect, the data transmission method according to the third aspect further includes:
acquiring first task information corresponding to the first data transmission task, wherein the first task information at least comprises a first data source identifier and a first data identifier;
the collecting data from the data source comprises:
linking a first data source corresponding to the first data source identifier according to the first data source identifier;
and reading first data corresponding to the first data identifier from the first data source according to the first data identifier.
In a fifth aspect, according to the data transmission method in the fourth aspect, the first task information further includes: a first message queue identity;
before organizing the content of the data as a message in a target message queue, the method further comprises:
and selecting a first message queue corresponding to the first message queue identification from a plurality of preconfigured message queues.
According to a sixth aspect, in the data transmission method of the third aspect, messages corresponding to different data transmission tasks in the target message queue are distinguished by different topics; the respectively establishing the transmission channels of the target message queue and the plurality of designated data storage systems comprises:
respectively establishing a target message queue and transmission channels of the appointed data storage systems according to different themes of the target message queue and aiming at different data transmission tasks;
the transmitting messages of the target message queue in parallel to the plurality of data storage systems through the transmission channel comprises:
and aiming at any data transmission task, transmitting the messages corresponding to the data transmission task in the target message queue to each data storage system in parallel through the transmission channel corresponding to the data transmission task.
In a seventh aspect, according to the data transmission method of the sixth aspect, the target message queue further includes a second message queue corresponding to a second data transmission task; the data further comprises second data corresponding to the second data transmission task.
In an eighth aspect, according to the data transmission method in the seventh aspect, the first message queue is the same as the second message queue; the messages in the first message queue include: a first message of content organization of the first data, and a second message of content organization of the second data; and the first theme corresponding to the first message in the first message queue is different from the second theme corresponding to the second message.
In a ninth aspect, according to the data transmission method in the eighth aspect, the respectively establishing, according to different subjects of the target message queue, transmission channels between the target message queue and each specified data storage system for different data transmission tasks includes:
establishing a first message queue and first transmission channels of a plurality of data storage systems respectively according to the first theme, and establishing a first message queue and second transmission channels of the plurality of data storage systems respectively according to the second theme;
the parallel transmission of the messages corresponding to the data transmission tasks in the target message queue to each data storage system through the transmission channels corresponding to the data transmission tasks aiming at any data transmission task comprises the following steps:
and transmitting the first messages of the first message queue to the plurality of data storage systems in parallel through the established first transmission channel, and transmitting the second messages of the first message queue to the plurality of data storage systems in parallel through the established second transmission channel.
Tenth aspect, according to the data transmission method of the seventh aspect, the first message queue is different from the second message queue; the messages in the first message queue include: a first message of a content organization of the first data; the messages in the second message queue include: a second message of a content organization of the second data; and the first theme corresponding to the first message in the first message queue is different from the second theme corresponding to the second message in the second message queue.
In an eleventh aspect, according to the data transmission method in the tenth aspect, the respectively establishing, according to different subjects of the target message queue, transmission channels between the target message queue and each specified data storage system for different data transmission tasks includes:
respectively establishing a first message queue and first transmission channels of a plurality of data storage systems according to the first theme, and respectively establishing a second message queue and second transmission channels of the plurality of data storage systems according to the second theme;
the parallel transmission of the messages corresponding to the data transmission tasks in the target message queue to each data storage system through the transmission channels corresponding to the data transmission tasks aiming at any data transmission task comprises the following steps:
and transmitting the first messages of the first message queue to the plurality of data storage systems in parallel through the established first transmission channel, and transmitting the second messages of the second message queue to the plurality of data storage systems in parallel through the established second transmission channel.
In a twelfth aspect, an embodiment of the present invention further provides a data transmission apparatus, including:
an acquisition module for acquiring data from a data source;
the message organization module is used for organizing the content of the data into messages in a target message queue;
the channel establishing module is used for respectively establishing the target message queue and the transmission channels of the plurality of specified data storage systems;
and the transmission module is used for transmitting the messages of the target message queue to the plurality of data storage systems in parallel through the transmission channel.
In a thirteenth aspect, an embodiment of the present invention further provides a client, where the client is configured to execute the data transmission method in any one of the first aspect to the eleventh aspect.
In a fourteenth aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes at least one memory and at least one processor, where the memory stores a program, and the processor calls the program, where the program is configured to:
collecting data from a data source;
organizing the content of the data into messages in a target message queue;
respectively establishing the target message queue and transmission channels of a plurality of specified data storage systems;
and transmitting the messages of the target message queue to the plurality of data storage systems in parallel through the transmission channel.
A fifteenth aspect, the electronic device of the fourteenth aspect, the program for organizing content of the data into messages in a target message queue, comprising:
packaging the content of the data into event data according to rows;
and taking an event data as a message, and organizing the message in a target message queue in a queue form.
A sixteenth aspect, according to the electronic device of the fourteenth or fifteenth aspect, the target message queue includes a first message queue corresponding to a first data transmission task; the data comprises first data corresponding to the first data transmission task.
A seventeenth aspect, the electronic device according to the sixteenth aspect, wherein the program is further configured to:
acquiring first task information corresponding to the first data transmission task, wherein the first task information at least comprises a first data source identifier and a first data identifier;
the program for collecting data from a data source, comprising:
linking a first data source corresponding to the first data source identifier according to the first data source identifier;
and reading first data corresponding to the first data identifier from the first data source according to the first data identifier.
Eighteenth, the electronic device of seventeenth, wherein the first task information further comprises: a first message queue identity; the program is also for:
before organizing the content of the data into messages in a target message queue, selecting a first message queue corresponding to the first message queue identification from a plurality of pre-configured message queues.
Nineteenth aspect, according to the electronic device of the sixteenth aspect, messages corresponding to different data transmission tasks in the target message queue are distinguished by different topics; the program is configured to establish the target message queue and transmission channels of a plurality of specified data storage systems, and specifically includes:
respectively establishing a target message queue and transmission channels of the appointed data storage systems according to different themes of the target message queue and aiming at different data transmission tasks;
the program is configured to transmit the messages in the target message queue to the plurality of data storage systems in parallel through the transmission channel, and specifically includes:
and aiming at any data transmission task, transmitting the messages corresponding to the data transmission task in the target message queue to each data storage system in parallel through the transmission channel corresponding to the data transmission task.
A twentieth aspect of the present invention provides the electronic device according to the nineteenth aspect, wherein the target message queue further includes a second message queue corresponding to a second data transmission task; the data further comprises second data corresponding to the second data transmission task.
A twenty-first aspect, the electronic device of the twentieth aspect, wherein the first message queue is the same as the second message queue; the messages in the first message queue include: a first message of content organization of the first data, and a second message of content organization of the second data; and the first theme corresponding to the first message in the first message queue is different from the second theme corresponding to the second message.
A twenty-second aspect of the electronic device according to the twenty-first aspect, where the program is configured to respectively establish transmission channels between the target message queue and each specified data storage system for different data transmission tasks according to different topics of the target message queue, and the program includes:
establishing a first message queue and first transmission channels of a plurality of data storage systems respectively according to the first theme, and establishing a first message queue and second transmission channels of the plurality of data storage systems respectively according to the second theme;
the program is used for transmitting the messages corresponding to the data transmission tasks in the target message queue to each data storage system in parallel through the transmission channels corresponding to the data transmission tasks aiming at any data transmission task, and the program comprises the following steps:
and transmitting the first messages of the first message queue to the plurality of data storage systems in parallel through the established first transmission channel, and transmitting the second messages of the first message queue to the plurality of data storage systems in parallel through the established second transmission channel.
A twenty-third aspect, the electronic device of the twentieth aspect, wherein the first message queue is different from the second message queue; the messages in the first message queue include: a first message of a content organization of the first data; the messages in the second message queue include: a second message of a content organization of the second data; and the first theme corresponding to the first message in the first message queue is different from the second theme corresponding to the second message in the second message queue.
Twenty-fourth, the electronic device according to the twenty-third, wherein the program is configured to respectively establish transmission channels between the target message queue and each specified data storage system for different data transmission tasks according to different topics of the target message queue, and the program includes:
respectively establishing a first message queue and first transmission channels of a plurality of data storage systems according to the first theme, and respectively establishing a second message queue and second transmission channels of the plurality of data storage systems according to the second theme;
the program is used for transmitting the messages corresponding to the data transmission tasks in the target message queue to each data storage system in parallel through the transmission channels corresponding to the data transmission tasks aiming at any data transmission task, and the program comprises the following steps:
and transmitting the first messages of the first message queue to the plurality of data storage systems in parallel through the established first transmission channel, and transmitting the second messages of the second message queue to the plurality of data storage systems in parallel through the established second transmission channel.
In a twenty-fifth aspect, an embodiment of the present invention further provides a storage medium storing a program for executing the data transmission method according to any one of the first to eleventh aspects.

Claims (25)

1. A method of data transmission, comprising:
collecting data from a data source;
organizing the content of the data into messages in a target message queue;
respectively establishing the target message queue and transmission channels of a plurality of specified data storage systems;
transmitting messages of a target message queue in parallel to the plurality of data storage systems via the transmission channel, comprising: transmitting the messages in the target message queue to the first data storage system through a third transmission channel established between the target message queue and the first data storage system, and transmitting the messages in the target message queue to the second data storage system through a fourth transmission channel established between the target message queue and the second data storage system;
the messages corresponding to different data transmission tasks in the target message queue are distinguished by different themes; the establishing of the transmission channels of the message queue and the plurality of designated data storage systems respectively comprises: and respectively establishing a transmission channel between the target message queue and each appointed data storage system aiming at different data transmission tasks according to different themes of the target message queue.
2. The data transmission method of claim 1, wherein the organizing the content of the data into messages in a target message queue comprises:
packaging the content of the data into event data according to rows;
and taking an event data as a message, and organizing the message in a target message queue in a queue form.
3. The data transmission method according to claim 1 or 2, wherein the target message queue comprises a first message queue corresponding to a first data transmission task; the data comprises first data corresponding to the first data transmission task.
4. The data transmission method according to claim 3, further comprising:
acquiring first task information corresponding to the first data transmission task, wherein the first task information at least comprises a first data source identifier and a first data identifier;
the collecting data from the data source comprises:
linking a first data source corresponding to the first data source identifier according to the first data source identifier;
and reading first data corresponding to the first data identifier from the first data source according to the first data identifier.
5. The data transmission method according to claim 4, wherein the first task information further comprises: a first message queue identity;
before organizing the content of the data as a message in a target message queue, the method further comprises:
and selecting a first message queue corresponding to the first message queue identification from a plurality of preconfigured message queues.
6. The data transmission method according to claim 3, wherein the transmitting messages of the target message queue to the plurality of data storage systems in parallel through the transmission channel comprises:
and aiming at any data transmission task, transmitting the messages corresponding to the data transmission task in the target message queue to each data storage system in parallel through the transmission channel corresponding to the data transmission task.
7. The data transmission method according to claim 6, wherein the target message queue further comprises a second message queue corresponding to a second data transmission task; the data further comprises second data corresponding to the second data transmission task.
8. The data transmission method according to claim 7, wherein the first message queue is the same as the second message queue; the messages in the first message queue include: a first message of content organization of the first data, and a second message of content organization of the second data; and the first theme corresponding to the first message in the first message queue is different from the second theme corresponding to the second message.
9. The data transmission method according to claim 8, wherein the establishing, according to different topics of the target message queue, transmission channels between the target message queue and each specified data storage system for different data transmission tasks respectively comprises:
establishing a first message queue and first transmission channels of a plurality of data storage systems respectively according to the first theme, and establishing a first message queue and second transmission channels of the plurality of data storage systems respectively according to the second theme;
the parallel transmission of the messages corresponding to the data transmission tasks in the target message queue to each data storage system through the transmission channels corresponding to the data transmission tasks aiming at any data transmission task comprises the following steps:
and transmitting the first messages of the first message queue to the plurality of data storage systems in parallel through the established first transmission channel, and transmitting the second messages of the first message queue to the plurality of data storage systems in parallel through the established second transmission channel.
10. The data transmission method according to claim 7, wherein the first message queue is different from the second message queue; the messages in the first message queue include: a first message of a content organization of the first data; the messages in the second message queue include: a second message of a content organization of the second data; and the first theme corresponding to the first message in the first message queue is different from the second theme corresponding to the second message in the second message queue.
11. The data transmission method according to claim 10, wherein the establishing, according to different topics of the target message queue, transmission channels between the target message queue and each specified data storage system for different data transmission tasks respectively comprises:
respectively establishing a first message queue and first transmission channels of a plurality of data storage systems according to the first theme, and respectively establishing a second message queue and second transmission channels of the plurality of data storage systems according to the second theme;
the parallel transmission of the messages corresponding to the data transmission tasks in the target message queue to each data storage system through the transmission channels corresponding to the data transmission tasks aiming at any data transmission task comprises the following steps:
and transmitting the first messages of the first message queue to the plurality of data storage systems in parallel through the established first transmission channel, and transmitting the second messages of the second message queue to the plurality of data storage systems in parallel through the established second transmission channel.
12. A data transmission apparatus, comprising:
an acquisition module for acquiring data from a data source;
the message organization module is used for organizing the content of the data into messages in a target message queue;
the channel establishing module is used for respectively establishing the target message queue and the transmission channels of the plurality of specified data storage systems;
the transmission module is used for transmitting the messages of the target message queue to the plurality of data storage systems in parallel through the transmission channel;
the messages corresponding to different data transmission tasks in the target message queue are distinguished by different themes; the establishing of the transmission channels of the message queue and the plurality of designated data storage systems respectively comprises: respectively establishing a target message queue and transmission channels of the appointed data storage systems according to different themes of the target message queue and aiming at different data transmission tasks;
the transmitting messages of the target message queue to the plurality of data storage systems in parallel through the transmission channel comprises: and transmitting the messages in the target message queue to the first data storage system through a third transmission channel established between the target message queue and the first data storage system, and transmitting the messages in the target message queue to the second data storage system through a fourth transmission channel established between the target message queue and the second data storage system.
13. A client, characterized in that the client is configured to perform the data transmission method of any of claims 1-11.
14. An electronic device, comprising at least one memory and at least one processor, the memory storing a program, the processor invoking the program, the program configured to:
collecting data from a data source;
organizing the content of the data into messages in a target message queue;
respectively establishing the target message queue and transmission channels of a plurality of specified data storage systems;
transmitting messages of a target message queue in parallel to the plurality of data storage systems via the transmission channel, comprising: transmitting the messages in the target message queue to the first data storage system through a third transmission channel established between the target message queue and the first data storage system, and transmitting the messages in the target message queue to the second data storage system through a fourth transmission channel established between the target message queue and the second data storage system;
the messages corresponding to different data transmission tasks in the target message queue are distinguished by different themes; the establishing of the transmission channels of the message queue and the plurality of designated data storage systems respectively comprises: and respectively establishing a transmission channel between the target message queue and each appointed data storage system aiming at different data transmission tasks according to different themes of the target message queue.
15. The electronic device of claim 14, wherein the program is configured to organize the content of the data as messages in a target message queue, comprising:
packaging the content of the data into event data according to rows;
and taking an event data as a message, and organizing the message in a target message queue in a queue form.
16. The electronic device of claim 14 or 15, wherein the target message queue comprises a first message queue corresponding to a first data transmission task; the data comprises first data corresponding to the first data transmission task.
17. The electronic device of claim 16, wherein the program is further configured to:
acquiring first task information corresponding to the first data transmission task, wherein the first task information at least comprises a first data source identifier and a first data identifier;
the program for collecting data from a data source, comprising:
linking a first data source corresponding to the first data source identifier according to the first data source identifier;
and reading first data corresponding to the first data identifier from the first data source according to the first data identifier.
18. The electronic device of claim 17, wherein the first task information further comprises: a first message queue identity; the program is also for:
before organizing the content of the data into messages in a target message queue, selecting a first message queue corresponding to the first message queue identification from a plurality of pre-configured message queues.
19. The electronic device according to claim 16, wherein the program is configured to transmit the messages in the target message queue to the plurality of data storage systems in parallel via the transmission channel, and specifically includes:
and aiming at any data transmission task, transmitting the messages corresponding to the data transmission task in the target message queue to each data storage system in parallel through the transmission channel corresponding to the data transmission task.
20. The electronic device of claim 19, wherein the target message queue further comprises a second message queue corresponding to a second data transmission task; the data further comprises second data corresponding to the second data transmission task.
21. The electronic device of claim 20, wherein the first message queue is the same as the second message queue; the messages in the first message queue include: a first message of content organization of the first data, and a second message of content organization of the second data; and the first theme corresponding to the first message in the first message queue is different from the second theme corresponding to the second message.
22. The electronic device according to claim 21, wherein the program is configured to establish transmission channels between the target message queue and each of the designated data storage systems for different data transmission tasks according to different topics of the target message queue, and includes:
establishing a first message queue and first transmission channels of a plurality of data storage systems respectively according to the first theme, and establishing a first message queue and second transmission channels of the plurality of data storage systems respectively according to the second theme;
the program is used for transmitting the messages corresponding to the data transmission tasks in the target message queue to each data storage system in parallel through the transmission channels corresponding to the data transmission tasks aiming at any data transmission task, and the program comprises the following steps:
and transmitting the first messages of the first message queue to the plurality of data storage systems in parallel through the established first transmission channel, and transmitting the second messages of the first message queue to the plurality of data storage systems in parallel through the established second transmission channel.
23. The electronic device of claim 20, wherein the first message queue is different from the second message queue; the messages in the first message queue include: a first message of a content organization of the first data; the messages in the second message queue include: a second message of a content organization of the second data; and the first theme corresponding to the first message in the first message queue is different from the second theme corresponding to the second message in the second message queue.
24. The electronic device according to claim 23, wherein the program is configured to establish transmission channels between the target message queue and each of the designated data storage systems for different data transmission tasks according to different topics of the target message queue, and includes:
respectively establishing a first message queue and first transmission channels of a plurality of data storage systems according to the first theme, and respectively establishing a second message queue and second transmission channels of the plurality of data storage systems according to the second theme;
the program is used for transmitting the messages corresponding to the data transmission tasks in the target message queue to each data storage system in parallel through the transmission channels corresponding to the data transmission tasks aiming at any data transmission task, and the program comprises the following steps:
and transmitting the first messages of the first message queue to the plurality of data storage systems in parallel through the established first transmission channel, and transmitting the second messages of the second message queue to the plurality of data storage systems in parallel through the established second transmission channel.
25. A storage medium characterized in that the storage medium stores a computer program which, when executed by a processor, executes a program of the data transmission method according to any one of claims 1 to 11.
CN201910704055.3A 2019-07-31 2019-07-31 Data transmission method, device, client, electronic equipment and storage medium Active CN110417898B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910704055.3A CN110417898B (en) 2019-07-31 2019-07-31 Data transmission method, device, client, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910704055.3A CN110417898B (en) 2019-07-31 2019-07-31 Data transmission method, device, client, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN110417898A CN110417898A (en) 2019-11-05
CN110417898B true CN110417898B (en) 2022-02-22

Family

ID=68364862

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910704055.3A Active CN110417898B (en) 2019-07-31 2019-07-31 Data transmission method, device, client, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN110417898B (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN204993390U (en) * 2015-10-07 2016-01-20 杭州锐达数字技术有限公司 A two transmission system for data collection station
CN106452819A (en) * 2015-08-13 2017-02-22 腾讯科技（深圳）有限公司 Data acquisition system and data acquisition method
CN106559269A (en) * 2016-11-29 2017-04-05 青岛国瑞信息技术有限公司 Towards redundant data collection and the Transmission system of electric power enterprise secondary protection
CN109063196A (en) * 2018-09-03 2018-12-21 拉扎斯网络科技（上海）有限公司 Data processing method, device, electronic equipment and computer readable storage medium
CN109104479A (en) * 2018-08-01 2018-12-28 福州大学 A kind of acquisition data flow transmission method and system based on multidiameter delay transmission technology
CN109143878A (en) * 2018-08-06 2019-01-04 广州供电局有限公司 The remote signalling message and SOE message treatment method in multi pass acquisition channel
CN109327351A (en) * 2018-09-12 2019-02-12 拉扎斯网络科技（上海）有限公司 Real-time collecting method, device, electronic equipment and the storage medium of daily record data
CN110071873A (en) * 2019-04-09 2019-07-30 华为技术有限公司 A kind of method, apparatus and relevant device sending data

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10331484B2 (en) * 2017-11-14 2019-06-25 Bank Of America Corporation Distributed data platform resource allocator

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106452819A (en) * 2015-08-13 2017-02-22 腾讯科技（深圳）有限公司 Data acquisition system and data acquisition method
CN204993390U (en) * 2015-10-07 2016-01-20 杭州锐达数字技术有限公司 A two transmission system for data collection station
CN106559269A (en) * 2016-11-29 2017-04-05 青岛国瑞信息技术有限公司 Towards redundant data collection and the Transmission system of electric power enterprise secondary protection
CN109104479A (en) * 2018-08-01 2018-12-28 福州大学 A kind of acquisition data flow transmission method and system based on multidiameter delay transmission technology
CN109143878A (en) * 2018-08-06 2019-01-04 广州供电局有限公司 The remote signalling message and SOE message treatment method in multi pass acquisition channel
CN109063196A (en) * 2018-09-03 2018-12-21 拉扎斯网络科技（上海）有限公司 Data processing method, device, electronic equipment and computer readable storage medium
CN109327351A (en) * 2018-09-12 2019-02-12 拉扎斯网络科技（上海）有限公司 Real-time collecting method, device, electronic equipment and the storage medium of daily record data
CN110071873A (en) * 2019-04-09 2019-07-30 华为技术有限公司 A kind of method, apparatus and relevant device sending data

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
基于 Storm 与 Hadoop 的日志数据实时处理研究;李洋;《中国优秀硕士学位论文全文数据库（电子期刊）信息科技辑》;20180215;第33-35页 *

Also Published As

Publication number Publication date
CN110417898A (en) 2019-11-05

Similar Documents

Publication Publication Date Title
CN108243120A (en) Business Stream transmission method, device and communication system based on flexible Ethernet
CN108683720A (en) A kind of container cluster service configuration method and device
CN109218133A (en) Network speed testing system, method, apparatus and computer readable storage medium
CN115065637B (en) Method and device for transmitting computing power resource information and electronic equipment
CN109656572A (en) Packaging method and device, computer equipment, the storage medium of installation kit
CN112788060A (en) Data packet transmission method and device, storage medium and electronic equipment
CN112492054A (en) Multiplexing method, device and equipment for UDP (user Datagram protocol) port
CN103109497B (en) For measuring the dynamic-configuration of the interconnect equipment of net neutral energy characteristic
CN111447143A (en) Business service data transmission method and device, computer equipment and storage medium
CN114510452A (en) SOC (System on chip) integration method and device and electronic equipment
CN106941522B (en) Lightweight distributed computing platform and data processing method thereof
CN107343037B (en) Data transmission method and device of distributed storage system
CN110417898B (en) Data transmission method, device, client, electronic equipment and storage medium
CN113452630B (en) Data merging method, data splitting method, device, equipment and storage medium
CN116010130B (en) Cross-card link aggregation method, device, equipment and medium for DPU virtual port
CN109150720A (en) Business chain message forwarding method, device, equipment and computer readable storage medium
CN116389323A (en) Throughput test method, device and storage medium
CN111756649A (en) Data transmission method, device, equipment and computer readable storage medium
US10148518B2 (en) Method and apparatus for managing computer system
CN112650710A (en) Data migration sending method and device, storage medium and electronic device
CN112291254A (en) Message processing method and device for reliable transaction
CN113676510A (en) Cloud storage method, system, equipment and storage medium
CN114500302B (en) ICE service arrangement method, ICE service arrangement device, terminal equipment and storage medium
CN112968794A (en) Network function chain deployment method, device, terminal device and storage medium
CN114567410B (en) Signal frame processing method and related equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant