CN115210694A

CN115210694A - Data transmission method and device

Info

Publication number: CN115210694A
Application number: CN202080097945.3A
Authority: CN
Inventors: 劳志毅
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd; Shenzhen Huantai Technology Co Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd; Shenzhen Huantai Technology Co Ltd
Priority date: 2020-03-06
Filing date: 2020-03-06
Publication date: 2022-10-18
Also published as: WO2021174537A1

Abstract

The embodiment of the application discloses a data transmission method and a device, which are applied to a subscription end device in a data transmission system, wherein the data transmission system also comprises a source end device, a consumption end device and a target end device, and a data set to be synchronized is acquired from the source end device; the data set to be synchronized is transmitted to the consumer terminal equipment in a two-stage submission mode, the consumer terminal equipment is instructed to hash the data set to be synchronized to a plurality of files of the WAL, and the files are transmitted to the target terminal equipment.

Description

Data transmission method and device

Technical Field

The present application relates to the field of electronic technologies, and in particular, to a data transmission method and apparatus.

Background

In the existing data synchronization technology, a storage service, such as a self mechanism of a structured query language (mysql), is relied on to ensure the orderliness in the data synchronization process, the mysql is a relational database management system, the relational database stores data in different tables, and the Structured Query Language (SQL) used by the mysql is the most common standardized language for accessing the database. In the synchronization process, the ordering of data synchronization is achieved by giving a globally unique ordering flag.

However, the characteristics of the storage service are excessively depended on, and the data transmission process depends on the global unique ordered mark, so that the input and the output of the data are single-thread synchronous, and the efficiency in the data transmission process is influenced.

Disclosure of Invention

The embodiment of the application provides a data transmission method and device, which can ensure that data is not lost and improve data transmission efficiency in a data synchronous transmission process.

In a first aspect, an embodiment of the present application provides a data transmission method, which is applied to a subscriber device in a data transmission system, where the data transmission system further includes a source device, a consumer device, and a target device, and the method includes:

acquiring a data set to be synchronized from the source device;

and transmitting the data set to be synchronized to the consumer terminal equipment in a two-stage submission mode, instructing the consumer terminal equipment to hash the data set to be synchronized to a plurality of files of the WAL, and transmitting the plurality of files to the target terminal equipment.

In a second aspect, an embodiment of the present application provides a data transmission method, which is applied to a consumer device in a data transmission system, where the data transmission system further includes a source device, a subscriber device, and a target device, and the method includes:

receiving a data set to be synchronized transmitted by the subscriber device in a two-stage submission manner, wherein the data set to be synchronized is data acquired by the subscriber device from the source device;

hashing the data set to be synchronized into a plurality of files of the WAL;

and transmitting the files to the target end equipment.

In a third aspect, an embodiment of the present application provides a data transmission apparatus, which is applied to a subscription end device in a data transmission system, where the data transmission system further includes a source end device, a consumer end device, and a target end device, and the data transmission apparatus includes:

a first obtaining unit, configured to obtain a data set to be synchronized from the source device;

the first transmission unit is used for transmitting the data set to be synchronized to the consumer terminal equipment in a two-stage submission mode, instructing the consumer terminal equipment to hash the data set to be synchronized to a plurality of files of the WAL, and transmitting the plurality of files to the target terminal equipment.

In a fourth aspect, an embodiment of the present application provides a data transmission apparatus, which is applied to a consumer device in a data transmission system, where the data transmission system further includes a source device, a subscriber device, and a target device, and the data transmission apparatus includes:

a receiving unit, configured to receive a to-be-synchronized data set transmitted by the subscriber device in a two-stage submission manner, where the to-be-synchronized data set is data acquired by the subscriber device from the source device;

the hash unit is used for hashing the data set to be synchronized into a plurality of files of the WAL;

and the second transmission unit is used for transmitting the files to the target end equipment.

In a fifth aspect, an embodiment of the present application provides a subscriber device, including a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the first aspect of the embodiment of the present application.

In a sixth aspect, an embodiment of the present application provides a consumer device, including a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for executing the steps in the first aspect of the embodiment of the present application.

In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program for electronic data exchange, where the computer program enables a computer to perform some or all of the steps described in the first aspect of the embodiment of the present application.

In an eighth aspect, embodiments of the present application provide a computer program product, where the computer program product includes a non-transitory computer-readable storage medium storing a computer program, where the computer program is operable to cause a computer to perform some or all of the steps as described in the first aspect of embodiments of the present application. The computer program product may be a software installation package.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

Fig. 1A is a schematic architecture diagram of a data transmission system for implementing a data transmission method according to an embodiment of the present application;

fig. 1B is a schematic flowchart of a data transmission method disclosed in an embodiment of the present application;

fig. 2 is a schematic flow chart of another data transmission method disclosed in the embodiments of the present application;

fig. 3 is a schematic flowchart of another data transmission method disclosed in the embodiment of the present application;

fig. 4A is a schematic structural diagram of another subscriber device disclosed in an embodiment of the present application;

FIG. 4B is a schematic diagram of another consumer device disclosed in an embodiment of the present application;

fig. 5 is a schematic structural diagram of a data transmission device disclosed in an embodiment of the present application;

fig. 6A is a schematic structural diagram of another data transmission device disclosed in an embodiment of the present application;

fig. 6B is a modified structure of the data transmission device shown in fig. 6A disclosed in the embodiment of the present application.

Detailed Description

In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.

The terms "first," "second," and the like in the description and claims of the present application and in the above-described drawings are used for distinguishing between different objects and not for describing a particular order. Furthermore, the terms "include" and "have," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.

In order to better understand the technical solutions described in the present invention, the following explains the technical terms related to the embodiments of the present invention:

mysql: the system is a relational database management system, and the relational database stores data in different tables.

Two-phase commit (two phase commit,2 PC): an algorithm for keeping consistency of all nodes under a distributed system architecture when transaction submission is carried out.

Write-Ahead Logging (WAL): an efficient log algorithm, for a non-memory database, disk I/O operations are a big bottleneck of database efficiency. Under the same data volume, when the database system adopting WAL submits the transaction, the disk write operation is only about half of the traditional rollback log, thereby greatly improving the efficiency of the disk I/O operation of the database and further improving the performance of the database.

The following describes embodiments of the present application in detail.

The data transmission method provided by the embodiment of the application is suitable for a cross-machine room data synchronous transmission scene. Fig. 1A is a schematic structural diagram of a data transmission system for implementing a data transmission method according to an embodiment of the present invention. The data transmission system may include a source device 1001, a subscriber device 1002, a consumer device 1003, and a target device 1004, where the source device is a device of a data publisher, the subscriber device is a device of a data subscriber, the consumer device is a device of a data consumer, the target device is a device of a data synchronous storage party, the source device is connected to the subscriber device, the subscriber device is further connected to the consumer device, the consumer device is further connected to the target device, and the source device 1001, the subscriber device 1002, the consumer device 1003, and the target device 1004 correspond to different computer rooms respectively. When the source device has a data set to be synchronized, which needs to be synchronously transmitted, the remote device can transmit the data set to be synchronized to the subscriber device, the subscriber device synchronously transmits the data set to be synchronized to the consumer device in a two-stage submission mode, and then the consumer device hashes the data set to be synchronized to a plurality of files of the WAL and transmits the files to the target device, so that synchronous data transmission across machine rooms can be realized, and data synchronization among different machine rooms in a plurality of places can be ensured. The data set to be synchronized may include at least one data to be synchronized.

Referring to fig. 1B, fig. 1B is a schematic flowchart of a data transmission method disclosed in an embodiment of the present application, and is applied to a subscriber device in a data transmission system, where the data transmission system further includes a source device, a consumer device, and a target device, and the data transmission method includes the following steps:

101. and acquiring a data set to be synchronized from the source end device.

In this embodiment of the present application, when there is data to be synchronized that needs to be synchronously transmitted in a remote device, the remote device may transmit a data set to be synchronized, which includes at least one data to be synchronized, to a subscribing device, so that the subscribing device may obtain the data set to be synchronized from a source device.

Optionally, in the step 101, the acquiring the data set to be synchronized from the source device may include the following steps:

and acquiring the data set to be synchronized transmitted by the source end equipment through a single thread.

When data synchronization transmission is performed, each piece of data to be synchronized in the data set to be synchronized carries a sequence identifier used for representing a transmission sequence, so that each piece of data to be synchronized transmitted by the source end device can be acquired in an ordered manner according to the sequence identifier corresponding to each piece of data to be synchronized in the data set to be synchronized. Specifically, the data set to be synchronized transmitted by the source end device is acquired through the single thread, and the data to be synchronized can be acquired according to the time sequence represented by the sequence identifier.

102. And transmitting the data set to be synchronized to the consumer terminal equipment in a two-stage submission mode, instructing the consumer terminal equipment to hash the data set to be synchronized to a plurality of files of the WAL, and transmitting the plurality of files to the target terminal equipment.

The method comprises the steps that a subscriber device and a consumer device can determine whether own operation is successful or not, but whether operation of other devices is successful or not cannot be determined, through a two-stage submission mode, the subscriber device can send an inquiry message whether sending a to-be-synchronized data set can be executed or not to the consumer device in a first stage, if the subscriber device receives a feedback message from the consumer device within preset time, the subscriber device enters a second stage, in the second stage, the subscriber device transmits the to-be-synchronized data set to the consumer device, and if the subscriber device does not receive the feedback message from the consumer device within preset time, the subscriber device sends a rollback (rollback) message to the consumer device, so that the to-be-synchronized data set can be guaranteed to be successfully transmitted to the consumer device. The subscriber device may instruct the consumer device to hash the data set to be synchronized into a plurality of files of the WAL, and transmit the plurality of files to the target device, so that the efficiency of transmitting the synchronized data may be improved by transmitting the data set to be synchronized to the plurality of files of the WAL.

It can be seen that, in the data transmission method described in the embodiment of the present application, a set of data to be synchronized is obtained from a source device; the data to be synchronized is transmitted to the consumption end equipment in a two-stage submission mode, the consumption end equipment is instructed to hash the data to be synchronized to the files of the WAL, and the files are transmitted to the target end equipment.

Consistent with the above, fig. 2 is a schematic flow chart of a data transmission method disclosed in the embodiment of the present application. The data transmission method is applied to consumer-side equipment in a data transmission system, the data transmission system also comprises source-side equipment, subscriber-side equipment and target-side equipment, and the data transmission method comprises the following steps:

201. and receiving a data set to be synchronized transmitted by the subscriber device in a two-stage submission manner, wherein the data set to be synchronized is data acquired by the subscriber device from the source device.

The method comprises the steps that a subscriber device and a consumer device can determine whether own operation is successful, but whether operation of other devices is successful cannot be determined, through a two-stage submission mode, the subscriber device can send an inquiry message whether sending of a data set to be synchronized can be executed to the consumer device in a first stage, if the subscriber device receives a feedback message from the consumer device within a preset time, the second stage is entered, the subscriber device transmits the data set to be synchronized to the consumer device in the second stage, and if the subscriber device does not receive the feedback message from the consumer device within the preset time, the subscriber device sends a rollback (rollback) message to the consumer device, so that the consumer device can be guaranteed to successfully receive the data set to be synchronized.

202. And hashing the data set to be synchronized into a plurality of files of the WAL.

The WAL can improve the efficiency of non-memory data and ensure the integrity of the data. Therefore, the WAL is used for storing the data to be synchronized in the data set to be synchronized, the data set to be synchronized is hashed into a plurality of files of the WAL, and then the plurality of files in the WAL are transmitted to the target end device, so that the transmission efficiency of the synchronized data can be improved. Specifically, data in a data set to be synchronized can be written into a plurality of files through a plurality of data transmission channels, so that the input and output utilization rate of a disk can be increased, and the file writing rate is improved.

Optionally, the step 202 of hashing the data set to be synchronized into a plurality of files of the WAL may include the following steps:

21. dividing the data to be synchronized into a plurality of groups according to a plurality of routing information corresponding to the data to be synchronized, wherein the data to be synchronized in each group corresponds to the same routing information;

22. and writing the data to be synchronized of each group in the plurality of groups into the file corresponding to the group in the WAL to obtain a plurality of files.

The routing information may be a device identifier of a routing device corresponding to the data to be synchronized, the data to be synchronized having the same routing information may be divided into one group according to the routing information of the data to be synchronized, then the data to be synchronized of each group is written into a file corresponding to the group, and then the data to be synchronized of a plurality of groups may be written into a plurality of files, and the plurality of groups correspond to the plurality of files one to one. Therefore, the number of the files can be dynamically adjusted according to the routing information, the number of the files is different, and the transmission rates of the data to be synchronized are also different, so that the transmission rate of the data set to be synchronized can be dynamically adjusted.

Optionally, in the step 22, writing the data to be synchronized in each of the plurality of packets into the file corresponding to the packet in the WAL to obtain a plurality of files, which may include the following steps:

2201. if the number of the data types of all data to be synchronized in a group i is larger than a preset numerical value, dividing all the data to be synchronized in the group i into a plurality of sub-groups, wherein the data to be synchronized in each sub-group corresponds to the same data type, and the group i is any one of the plurality of groups;

2202. and writing the data to be synchronized of each sub-packet in the plurality of sub-packets of the packet i into the file corresponding to the sub-packet in the WAL to obtain a plurality of files.

For any one of the multiple groups, if the number of the data types of all the data to be synchronized in the group i is greater than a preset value, it indicates that the number of the data types in the group i is greater, and the number of the data in the group i is greater, and further, all the data to be synchronized in the group i can be divided into multiple different sub-groups according to the data types, specifically, the data to be synchronized with the same data type in the group i is divided into the same sub-group, and further, the data to be synchronized of each sub-group is written into a file corresponding to the sub-group, so that the data to be synchronized of the same sub-group can be written into one file. Therefore, the number of the files can be dynamically adjusted according to the data types, and when the data types in the group i are more, the number of the corresponding files is more, the transmission rates of the data to be synchronized are different, so that the transmission rate of the data set to be synchronized can be dynamically adjusted and adjusted.

Optionally, the packet i includes m pieces of data to be synchronized, and in this embodiment of the present application, the method may further include the following steps:

obtaining the type identifiers of the m data to be synchronized in the group i to obtain m type identifiers;

and counting the m type identifications to obtain the number n of the data types of all the data to be synchronized in the group i, wherein n is less than or equal to m.

Before step 2201, the number n of data types of all data to be synchronized in the group i may also be determined, specifically, type identifiers of m data to be synchronized in the group i may be obtained to obtain m type identifiers, each data to be synchronized corresponds to one type identifier, different data to be synchronized may correspond to the same type identifier or different type identifiers, and further, the number n of the type identifiers may be counted.

Optionally, in the step 2202, writing the data to be synchronized in each of the multiple sub-packets of the packet i into the file corresponding to the sub-packet in the WAL to obtain multiple files, which may include the following steps:

and writing the data to be synchronized of each sub-packet in the plurality of sub-packets of the packet i into the file in the WAL according to a file sequential writing mode to obtain a plurality of files.

In the embodiment of the application, when the data to be synchronized is written into the file, a large amount of disk addressing time can be avoided according to the mode of writing the file in sequence, so that the file writing speed can be improved.

203. And transmitting the files to the target end equipment.

The consumption end equipment can transmit a plurality of files to the target end equipment, and the consistency of the data set to be synchronized at the data source end and the data set to be synchronized transmitted to the target end equipment is ensured through two-stage submission, so that the data set to be synchronized is not lost in the transmission process.

Optionally, in step 203, the transmitting the plurality of files to the target device may include the following steps:

and transmitting the plurality of files to the target end equipment through a plurality of asynchronous threads.

The multiple files are transmitted to the target end equipment through multiple asynchronous threads, so that the CPU utilization rate of the consumption end equipment can be improved, and the transmission rate of the multiple files can be improved.

It can be seen that, in the data transmission method described in the embodiment of the present application, a to-be-synchronized data set transmitted by a subscriber device is received in a two-stage submission manner, where the to-be-synchronized data set is data obtained by the subscriber device from a source device; hashing a data set to be synchronized into a plurality of files of the WAL; and transmitting the data sets to be synchronized to the consumption end equipment in a two-stage submission mode, so that the successful transmission of the data sets to be synchronized is ensured, and the data sets to be synchronized are transmitted to the files of the WAL, so that the data synchronization transmission efficiency is improved.

Consistent with the above, fig. 3 is a schematic flow chart of a data transmission method disclosed in the embodiment of the present application. Applied to the data transmission system shown in fig. 1A, the data transmission system may include a source device, a subscriber device, a consumer device, and a target device, and the data transmission method includes the following steps:

301. the method comprises the steps that a source end device sends a data set to be synchronized to a subscription end device through a single thread, wherein the data set to be synchronized comprises a plurality of data to be synchronized, and each data to be synchronized corresponds to one piece of routing information.

302. And the subscribing terminal equipment transmits the data set to be synchronized to the consuming terminal equipment in a two-stage submission mode.

303. And the consumer terminal equipment divides the data to be synchronized into a plurality of groups according to a plurality of routing information corresponding to the data to be synchronized, wherein the data to be synchronized in each group corresponds to the same routing information.

304. And the consumer terminal equipment writes the data to be synchronized of each group in the plurality of groups into the file corresponding to the group in the WAL according to a mode of sequentially writing the file to obtain a plurality of files.

305. And the consumer end equipment transmits the files to the target end equipment through a plurality of asynchronous threads.

The detailed descriptions of steps 301 to 305 may refer to the corresponding descriptions of the data transmission methods described in fig. 1B and fig. 2, and are not repeated herein.

It can be seen that, in the data transmission method described in this embodiment of the present application, a source device sends a set of data to be synchronized to a subscriber device through a single thread, where the set of data to be synchronized includes multiple sets of data to be synchronized, each set of data to be synchronized corresponds to one piece of routing information, the subscriber device transmits the set of data to be synchronized to a consumer device in a two-stage submission manner, the consumer device divides the multiple sets of data to be synchronized into multiple groups according to the multiple pieces of routing information corresponding to the multiple sets of data to be synchronized, the data to be synchronized in each group corresponds to the same piece of routing information, the consumer device writes the data to be synchronized in each group of the multiple groups into a file corresponding to the group in a WAL in a sequential file writing manner, so as to obtain multiple files, and the consumer device transmits the multiple files to a target device through multiple asynchronous threads, so that the set of data to be synchronized is transmitted to the consumer device in the two-stage submission manner, thereby ensuring that the set of data to be synchronized is successfully transmitted, and the set of data to be synchronized is transmitted through the multiple files in the WAL, thereby improving data synchronization transmission efficiency.

Referring to fig. 4A, fig. 4A is a schematic structural diagram of a subscriber device according to an embodiment of the present disclosure, and as shown in the drawing, the subscriber device includes a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the following steps:

acquiring a data set to be synchronized from the source end device;

and transmitting the data set to be synchronized to the consumer terminal equipment in a two-stage submission mode, instructing the consumer terminal equipment to hash the data set to be synchronized to a plurality of files of the pre-written log WAL, and transmitting the plurality of files to the target terminal equipment.

In one possible example, in the aspect of obtaining the data set to be synchronized from the source device, the program is specifically configured to execute the following steps:

Referring to fig. 4B, fig. 4B is a schematic structural diagram of a consumer device according to an embodiment of the present disclosure, as shown, the consumer device includes a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the memory and configured to be executed by the processor, and the programs include instructions for performing the following steps:

hashing the data set to be synchronized into a plurality of files of the WAL;

and transmitting the plurality of files to the target end equipment.

In one possible example, the set of data to be synchronized includes a plurality of data to be synchronized, each of the data to be synchronized corresponds to a piece of routing information, and in the hashing the set of data to be synchronized to a plurality of files of the WAL, the program includes instructions for performing the following steps:

dividing the data to be synchronized into a plurality of groups according to a plurality of routing information corresponding to the data to be synchronized, wherein the data to be synchronized in each group corresponds to the same routing information;

and writing the data to be synchronized of each group in the plurality of groups into a file corresponding to the group in a pre-written log WAL to obtain a plurality of files.

In one possible example, in the writing of the data to be synchronized of each packet in the plurality of packets into the file corresponding to the packet in the pre-write log WAL, a plurality of files are obtained, the program includes instructions for performing the following steps:

if the number of the data types of all data to be synchronized in a group i is larger than a preset numerical value, dividing all the data to be synchronized in the group i into a plurality of sub-groups, wherein the data to be synchronized in each sub-group corresponds to the same data type, and the group i is any one of the plurality of groups;

and writing the data to be synchronized of each sub-packet in the plurality of sub-packets of the packet i into the file corresponding to the sub-packet in the WAL to obtain a plurality of files.

In one possible example, the packet i includes m data to be synchronized, and the program further includes instructions for performing the following steps:

In one possible example, in the writing of the data to be synchronized of each of the plurality of sub-packets of the packet i to the file in the WAL to obtain a plurality of files, the program includes instructions for performing the following steps:

The above description has introduced the solution of the embodiment of the present application mainly from the perspective of the method-side implementation process. It is understood that the electronic device comprises corresponding hardware structures and/or software modules for performing the respective functions in order to realize the above-mentioned functions. Those of skill in the art will readily appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments provided herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiment of the present application, the electronic device may be divided into the functional units according to the method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit. It should be noted that, in the embodiment of the present application, the division of the unit is schematic, and is only one logic function division, and when the actual implementation is realized, another division manner may be provided.

Referring to fig. 5, fig. 5 is a schematic structural diagram of a data transmission apparatus, which is applied to a subscribing device in a data transmission system, where the data transmission system further includes a source device, a consuming device, and a target device, the data transmission apparatus includes a first obtaining unit 501 and a first transmission unit 502, where,

the first obtaining unit 501 is configured to obtain a data set to be synchronized from the source end device;

the first transmission unit 502 is configured to transmit the data set to be synchronized to the consumer device in a two-stage submission manner, instruct the consumer device to hash the data set to be synchronized to multiple files of the WAL, and transmit the multiple files to the target device.

Optionally, in the aspect of acquiring the to-be-synchronized data set from the source device, the first acquiring unit 501 is specifically configured to:

It can be seen that, in the data transmission apparatus described in this embodiment of the present application, a set of data to be synchronized is obtained from a source end device; the data to be synchronized is transmitted to the consumption end equipment in a two-stage submission mode, the consumption end equipment is instructed to hash the data to be synchronized to the files of the WAL, and the files are transmitted to the target end equipment.

It should be noted that the subscriber device described in the embodiment of the present application is presented in the form of a functional unit. The term "unit" as used herein should be understood in its broadest possible sense, and objects used to implement the functionality described in each "unit" may be, for example, an integrated circuit ASIC, a single circuit, a processor (shared, dedicated, or group) and memory that execute one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.

The first obtaining unit 501 and the first transmitting unit 502 may be a control circuit or a processor.

Referring to fig. 6A, fig. 6A is a schematic structural diagram of a data transmission apparatus, which is applied to a consumer device in a data transmission system, where the data transmission system further includes a source device, a subscribing device, and a target device, the data transmission apparatus includes a receiving unit 601, a hashing unit 602, and a second transmission unit 603, where,

the receiving unit 601 is configured to receive a set of data sets to be synchronized transmitted by the subscriber device in a two-stage submission manner, where the set of data sets to be synchronized is data obtained by the subscriber device from the source device;

the hashing unit 602, configured to hash the data set to be synchronized into multiple files of the WAL;

the second transmitting unit 603 is configured to transmit the plurality of files to the destination device.

Optionally, the data set to be synchronized includes multiple data to be synchronized, each of the data to be synchronized corresponds to one piece of routing information, and in the aspect of hashing the data set to be synchronized to multiple files of the WAL, the hashing unit 602 is specifically configured to:

Optionally, in the aspect that the data to be synchronized in each of the plurality of packets is written into a file corresponding to the packet in the pre-write log WAL to obtain a plurality of files, the hash unit is specifically configured to:

Optionally, the packet i includes m data to be synchronized, as shown in fig. 6B, where fig. 6B is a modified structure of the data transmission apparatus shown in fig. 6A, and compared with fig. 6A, the method may further include: a second acquisition unit 604 and a statistical unit 605, wherein,

the second obtaining unit 604 is configured to obtain the type identifiers of the m pieces of data to be synchronized in the group i, so as to obtain m pieces of type identifiers;

the counting unit 605 is configured to count the m type identifiers to obtain the number n of data types of all the data to be synchronized in the group i, where n is less than or equal to m.

Optionally, in the aspect that the data to be synchronized in each of the multiple sub-packets of the packet i is written into a file in the WAL to obtain multiple files, the hash unit is specifically configured to:

Optionally, in the aspect of transmitting the multiple files to the target end device, the second transmission unit is specifically configured to:

It can be seen that, in the data transmission apparatus described in the embodiment of the present application, a to-be-synchronized data set transmitted by a subscriber device is received in a two-stage submission manner, where the to-be-synchronized data set is data obtained by the subscriber device from a source device; hashing a data set to be synchronized into a plurality of files of the WAL; and transmitting the data sets to be synchronized to the consumption end equipment in a two-stage submission mode, so that the data sets to be synchronized are ensured to be successfully transmitted, and the data sets to be synchronized are transmitted to the files of the WAL, so that the data synchronization transmission efficiency is improved.

It should be noted that the consumer device described in the embodiments of the present application is presented in the form of a functional unit. The term "unit" as used herein is to be understood in its broadest possible sense, and objects used to implement the functions described by the respective "unit" may be, for example, an integrated circuit ASIC, a single circuit, a processor (shared, dedicated, or chipset) and memory that execute one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.

The receiving unit 601, the hashing unit 602, the second transmitting unit 603, the second obtaining unit 604, and the statistics unit 605 may be control circuits or processors.

Embodiments of the present application also provide a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, and the computer program enables a computer to execute part or all of the steps of any one of the data transmission methods as described in the above method embodiments.

Embodiments of the present application also provide a computer program product comprising a non-transitory computer readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the data transmission methods as recited in the above method embodiments.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art will recognize that the embodiments described in this specification are preferred embodiments and that acts or modules referred to are not necessarily required for this application.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one type of logical functional division, and other divisions may be realized in practice, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of some interfaces, devices or units, and may be an electric or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may be implemented in the form of a software program module.

The integrated units, if implemented in the form of software program modules and sold or used as stand-alone products, may be stored in a computer readable memory. Based on such understanding, the technical solution of the present application may be substantially implemented or a part of or all or part of the technical solution contributing to the prior art may be embodied in the form of a software product stored in a memory, and including several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned memory comprises: various media that can store program codes, such as a usb disk, a read-only memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.

Those skilled in the art will appreciate that all or part of the steps of the methods of the above embodiments may be implemented by a program, which is stored in a computer-readable memory, the memory including: flash disk, ROM, RAM, magnetic or optical disk, and the like.

The foregoing detailed description of the embodiments of the present application has been presented to illustrate the principles and implementations of the present application, and the above description of the embodiments is only provided to help understand the method and the core concept of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, the specific implementation manner and the application scope may be changed, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

A data transmission method, applied to a subscription end device in a data transmission system, where the data transmission system further includes a source end device, a consumer end device, and a target end device, and the method includes:

acquiring a data set to be synchronized from the source device;

and transmitting the data set to be synchronized to the consumer terminal equipment in a two-stage submission mode, instructing the consumer terminal equipment to hash the data set to be synchronized to a plurality of files of a pre-written log WAL, and transmitting the plurality of files to the target terminal equipment.
The method of claim 1, wherein the obtaining a set of data to be synchronized from the source device comprises:

and acquiring the data set to be synchronized transmitted by the source end equipment through a single thread.
A data transmission method, applied to a consumer device in a data transmission system, where the data transmission system further includes a source device, a subscriber device, and a target device, and the method includes:

receiving a data set to be synchronized transmitted by the subscriber device in a two-stage submission manner, wherein the data set to be synchronized is data acquired by the subscriber device from the source device;

hashing the data set to be synchronized into a plurality of files of the WAL;

and transmitting the files to the target end equipment.
The method as claimed in claim 3, wherein the data set to be synchronized includes a plurality of data to be synchronized, each of the data to be synchronized corresponds to a piece of routing information, and the hashing the data set to be synchronized into a plurality of files of the WAL includes:

dividing the data to be synchronized into a plurality of groups according to a plurality of routing information corresponding to the data to be synchronized, wherein the data to be synchronized in each group corresponds to the same routing information;

and writing the data to be synchronized of each group in the plurality of groups into the file corresponding to the group in the WAL to obtain a plurality of files.
The method as claimed in claim 4, wherein writing the data to be synchronized in each of the plurality of packets into the file corresponding to the packet in the WAL to obtain a plurality of files, comprises:

if the number of the data types of all data to be synchronized in a group i is larger than a preset numerical value, dividing all the data to be synchronized in the group i into a plurality of sub-groups, wherein the data to be synchronized in each sub-group corresponds to the same data type, and the group i is any one of the plurality of groups;

and writing the data to be synchronized of each sub-packet in the plurality of sub-packets of the packet i into the file corresponding to the sub-packet in the WAL to obtain a plurality of files.
The method of claim 5, wherein m data to be synchronized are included in the packet i, and the method further comprises:

obtaining the type identifiers of the m data to be synchronized in the group i to obtain m type identifiers;

and counting the m type identifications to obtain the number n of the data types of all the data to be synchronized in the group i, wherein n is less than or equal to m.
The method according to claim 5 or 6, wherein writing the data to be synchronized in each of the plurality of sub-packets of the packet i into the file in the WAL to obtain a plurality of files comprises:

and writing the data to be synchronized of each sub-packet in the plurality of sub-packets of the packet i into the file in the WAL according to a file sequential writing mode to obtain a plurality of files.
The method according to claim 7, wherein the transferring the plurality of files to the target device comprises:

and transmitting the plurality of files to the target end equipment through a plurality of asynchronous threads.
A data transmission apparatus, applied to a subscription end device in a data transmission system, where the data transmission system further includes a source end device, a consumer end device, and a target end device, and the data transmission apparatus includes:

a first obtaining unit, configured to obtain a data set to be synchronized from the source end device;

the first transmission unit is used for transmitting the data set to be synchronized to the consumer terminal equipment in a two-stage submission mode, instructing the consumer terminal equipment to hash the data set to be synchronized to a plurality of files of the WAL, and transmitting the plurality of files to the target terminal equipment.
The data transmission apparatus according to claim 9, wherein, in the acquiring the set of data to be synchronized from the source device, the first acquiring unit is specifically configured to:

and acquiring the data set to be synchronized transmitted by the source end equipment through a single thread.
A data transmission apparatus, applied to a consumer device in a data transmission system, where the data transmission system further includes a source device, a subscriber device, and a target device, and the data transmission apparatus includes:

a receiving unit, configured to receive a to-be-synchronized data set transmitted by the subscriber device in a two-stage submission manner, where the to-be-synchronized data set is data acquired by the subscriber device from the source device;

the hash unit is used for hashing the data set to be synchronized into a plurality of files of the WAL;

and the second transmission unit is used for transmitting the files to the target end equipment.
The data transmission apparatus according to any one of claims 7 to 9, wherein the data set to be synchronized includes a plurality of data to be synchronized, each of the data to be synchronized corresponds to one piece of routing information, and in the hashing the data set to be synchronized into the plurality of files of the WAL, the hashing unit is specifically configured to:

dividing the data to be synchronized into a plurality of groups according to a plurality of routing information corresponding to the data to be synchronized, wherein the data to be synchronized in each group corresponds to the same routing information;

and writing the data to be synchronized of each group in the plurality of groups into a file corresponding to the group in a pre-written log WAL to obtain a plurality of files.
The data transmission apparatus according to any one of claims 7 to 9, wherein in the writing of the data to be synchronized of each of the plurality of packets into the file corresponding to the packet in the pre-write log WAL to obtain a plurality of files, the hash unit is specifically configured to:

if the number of the data types of all data to be synchronized in a group i is larger than a preset numerical value, dividing all the data to be synchronized in the group i into a plurality of sub-groups, wherein the data to be synchronized in each sub-group corresponds to the same data type, and the group i is any one of the plurality of groups;

and writing the data to be synchronized of each sub-packet in the plurality of sub-packets of the packet i into the file corresponding to the sub-packet in the WAL to obtain a plurality of files.
The data transmission apparatus according to claim 11, wherein the packet i includes m data to be synchronized, the apparatus further includes a second acquisition unit and a statistic unit, wherein,

the second obtaining unit is configured to obtain the type identifiers of the m pieces of data to be synchronized in the group i to obtain m pieces of type identifiers;

and the counting unit is used for counting the m type identifications to obtain the number n of the data types of all the data to be synchronized in the group i, wherein n is less than or equal to m.
The data transmission apparatus according to claim 11, wherein in the writing of the data to be synchronized in each of the plurality of sub-packets of the packet i into the file in the WAL to obtain a plurality of files, the hashing unit is specifically configured to:

and writing the data to be synchronized of each sub-packet in the plurality of sub-packets of the packet i into the file in the WAL according to a file sequential writing mode to obtain a plurality of files.
The data transmission apparatus according to claim 11, wherein in the transmitting the plurality of files to the destination device, the second transmission unit is specifically configured to:

and transmitting the plurality of files to the target end equipment through a plurality of asynchronous threads.
A subscriber device comprising a processor, memory, a communication interface, and one or more programs stored in the memory and configured for execution by the processor, the programs including instructions for performing the steps of the method of claim 1 or 2.
A consumer device comprising a processor, a memory, a communication interface, and one or more programs stored in the memory and configured to be executed by the processor, the programs comprising instructions for performing the steps of the method of any of claims 3-8.
A computer-readable storage medium, characterized in that a computer program for electronic data exchange is stored, wherein the computer program causes a computer to perform the method according to any of the claims 1-8.
A computer program product, characterized in that the computer program product comprises a non-transitory computer readable storage medium storing a computer program, the computer program causing a computer to perform the method according to any one of claims 1-8.