CN111026735B

CN111026735B - Data transmission method, device, equipment and medium

Info

Publication number: CN111026735B
Application number: CN201911261581.3A
Authority: CN
Inventors: 胡婷婷
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2019-12-10
Filing date: 2019-12-10
Publication date: 2023-03-21
Anticipated expiration: 2039-12-10
Also published as: CN111026735A

Abstract

The data transmission method provided by the application comprises the following steps: acquiring a first plug-in according to the data type of the first database, wherein the first plug-in is used for reading first data in the first database; acquiring a second plug-in according to the data type of a second database, wherein the second plug-in is used for writing second data into the second database, and the second database and the first database are different types of databases; and acquiring a third plugin according to the data type of the first database and the data type of the second database, wherein the third plugin is used for converting the first data into second data and sending the second data to the second plugin. The application also provides a device, equipment and a medium, and the reading, format conversion and writing steps in the data transmission process of the heterogeneous databases are decoupled by setting the plug-in, so that the plug-in can be obtained according to the database type, the data transmission between the heterogeneous databases is realized, the code development amount is reduced, the upstream and downstream type support capability is expanded, and the data transmission realization efficiency is improved.

Description

Data transmission method, device, equipment and medium

Technical Field

The present invention relates to the field of computer technologies, and in particular, to a data transmission method, apparatus, device, and medium.

Background

With the continuous development of cloud computing capability and platform as a service (PaaS) capability of a cloud manufacturer, a data transmission function of a Data Transmission Service (DTS) is not only limited to a cloud requirement for serving a certain type of database, but also increasingly widely required in scenarios such as heterogeneous database migration. For example, many products require statistics on data changes of daily databases in order to generate daily reports, or to synchronize data between different databases, or to backup database local logs to a distributed storage platform, and so forth.

In the prior art, a data transmission task program is seriously coupled with a specific service, and lacks general capability in the aspect of heterogeneous database synchronization, so that when the service requirement is newly increased, both expansibility and reusability are problematic. Since the data transfer is coupled with the written code, any part of which changes, the transfer program needs to be rewritten, and the code reuse degree is low. For example, if a new requirement for transmitting data from MongoDB to MySQL or from Oracle to MySQL comes, a new program needs to be developed to implement the existing program for transmitting data from MySQL to MySQL.

Resulting in inefficient data transfer between disparate databases of different types.

Disclosure of Invention

In view of the above, in order to solve the above problems, the present invention provides the following technical solutions:

a method of data transmission, comprising:

acquiring a first plug-in according to the data type of a first database, wherein the first plug-in is used for reading first data in the first database;

acquiring a second plug-in according to the data type of a second database, wherein the second plug-in is used for writing second data into the second database, the second database and the first database are different types of databases, and the second data and the first data are different types of data;

and acquiring a third plugin according to the data type of the first database and the data type of the second database, wherein the third plugin is used for converting the first data read by the first plugin into the second data and sending the second data to the second plugin.

A data transmission apparatus comprising:

the system comprises a first acquisition unit, a second acquisition unit and a control unit, wherein the first acquisition unit is used for acquiring a first plug-in according to the data type of a first database, and the first plug-in is used for reading first data in the first database;

a second obtaining unit, configured to obtain a second plugin according to a data type of a second database, where the second plugin is configured to write second data into the second database, the second database and the first database are different types of databases, and the second data and the first data are different types of data;

a third obtaining unit, configured to obtain a third plugin according to the data type of the first database and the data type of the second database, where the third plugin is configured to convert the first data read by the first plugin obtained by the first obtaining unit into the second data and send the second data to the second plugin obtained by the second obtaining unit.

Optionally, the apparatus further comprises:

a fourth obtaining unit, configured to obtain a configuration file, where data types of the first database and the second database are recorded in the configuration file;

the first obtaining unit is further configured to:

acquiring the first plug-in according to the data type of the first database recorded in the configuration file acquired by the fourth acquisition unit;

the second obtaining unit is further configured to:

acquiring the second plug-in according to the data type of the second database recorded in the configuration file acquired by the fourth acquisition unit;

the third obtaining unit is further configured to:

and acquiring the third plug-in according to the data type of the first database and the data type of the second database recorded in the configuration file acquired by the fourth acquisition unit.

Optionally, the apparatus further comprises:

the first conversion unit is used for converting the first data into universal format data through the first plug-in, and the universal format data is a format which can be read by the second plug-in and the third plug-in;

the second conversion unit is used for converting the universal format data converted by the first conversion unit into the second data through the third plug-in;

and the writing unit is used for writing the second data converted by the second conversion unit into the second database through the second plug-in.

Optionally, the apparatus further includes a fifth obtaining unit, configured to: acquiring a Data Transmission Service (DTS) request;

the first obtaining unit is further configured to:

and starting to acquire the first plug-in according to the DTS request acquired by the fifth acquisition unit.

Optionally, the first conversion unit is further configured to:

starting a first thread through the first plug-in, wherein the first thread is used for exporting the first data from the first database;

converting the first data into the general format data and then putting the general format data into a first queue;

starting a second thread through the first plug-in;

calling the common format data from the first queue through the second thread;

and sending the data in the common format to the third plug-in through the first plug-in.

Optionally, the apparatus further includes a first execution unit, configured to:

marking a first version number of the first data acquired by the first plug-in, wherein the first version number is a globally unique version number;

sending the first version number to a progress controller, wherein the progress controller is used for recording the corresponding relation between the first data and the first version number;

when the first data is converted into the second data by the third plug-in and is written into the second database by the second plug-in, recording the data corresponding to the first version number as transmitted in the progress controller;

when the second plug-in writes the second data into the second database, acquiring a second version number corresponding to the second data from the progress controller, wherein the second version number is a globally unique version number;

if the second version number is recorded as transmitted in the progress controller, the second plug-in does not write the second data to the second database.

Optionally, the first execution unit is further configured to:

storing the data recorded as transmitted in the progress controller;

and storing the data which is not recorded as transmitted in the progress controller in a cache.

Optionally, the first database and the second database are both database systems, and the apparatus further includes a second execution unit, where the second execution unit is configured to:

acquiring a database primary key or a unique key of the first data through the first plug-in;

when the second plug-in writes second data into the second database, if the primary key or the unique key corresponding to the second database is written into the second database, the second plug-in does not write the second data any more.

Optionally, the first database and the second database are both a distributed file system or a message queue system, and the apparatus further includes a third execution unit, where the third execution unit is configured to:

attaching first offset position information of the first data to the first data by the first plug-in, the first offset position information being used to record an offset from a file start when the first data is stored on the disk;

converting the first data into second data through the third plug-in, wherein the corresponding relation between the second data and the first offset position is reserved;

and acquiring first offset position information of at least one second data through the second plug-in, and rejecting the second data corresponding to the repeated first offset information when the repeated first offset information exists.

Optionally, the apparatus further includes a fourth execution unit, configured to: and restarting the first plug-in and the second plug-in when detecting that the reading of the first plug-in is wrong or detecting that the writing of the second plug-in is wrong.

Optionally, the fourth execution unit is further configured to:

when the first plug-in and the second plug-in still can not work normally after being restarted, generating an error code;

acquiring a preset recovery strategy according to the error code, wherein the corresponding relation between the error code and the recovery strategy is a preset relation;

and recovering the transmission of the first plug-in and the second plug-in according to the recovery strategy.

Optionally, the apparatus further includes a fifth execution unit, configured to: when the first database is subjected to master-slave switching, first type information of the first database after the master-slave switching is acquired, wherein the first type information is used for recording the type of the first database;

a first plug-in corresponding to the first type information is obtained.

A computer device, the computer device comprising: an interaction device, an input/output (I/O) interface, a processor, and a memory having program instructions stored therein; the interaction device is used for acquiring an operation instruction input by a user; the processor is configured to execute program instructions stored in the memory to perform the method as described in any one of the above.

A computer readable storage medium comprising instructions which, when executed on a computer device, cause the computer device to perform a method as claimed in any preceding claim.

The data transmission method provided by the embodiment of the application comprises the following steps: acquiring a first plug-in according to the data type of the first database, wherein the first plug-in is used for reading first data in the first database; acquiring a second plug-in according to the data type of a second database, wherein the second plug-in is used for writing second data into the second database, the second database and the first database are different types of databases, and the second data and the first data are different types of data; and acquiring a third plugin according to the data type of the first database and the data type of the second database, wherein the third plugin is used for converting the first data read by the first plugin into second data and sending the second data to the second plugin. By setting the plug-in, the reading, format conversion and writing steps in the data transmission process of the heterogeneous database are decoupled, so that different plug-ins can be obtained according to different database types to realize data transmission between heterogeneous databases, the development amount of codes is reduced, the upstream and downstream type support capability is expanded, and the data transmission realization efficiency is improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

Fig. 1 is a flowchart of an embodiment of a data transmission method provided in an embodiment of the present application;

fig. 2 is a flowchart of another embodiment of a data transmission method according to an embodiment of the present application;

fig. 3 is a schematic diagram of another embodiment of a data transmission method according to an embodiment of the present application;

fig. 4 is a schematic diagram of another embodiment of a data transmission method according to an embodiment of the present application;

fig. 5 is a flowchart of another embodiment of a data transmission method according to an embodiment of the present application;

fig. 6 is a flowchart of another embodiment of a data transmission method according to an embodiment of the present application;

fig. 7 is a flowchart of another embodiment of a data transmission method according to an embodiment of the present application;

fig. 8 is a flowchart of another embodiment of a data transmission method according to an embodiment of the present application;

fig. 9 is a flowchart of another embodiment of a data transmission method provided in an embodiment of the present application;

fig. 10 is a flowchart of another embodiment of a data transmission method provided in an embodiment of the present application;

FIG. 11 is a schematic diagram of a computer device provided by an embodiment of the present application;

fig. 12 is a schematic diagram of a data transmission device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.

The terms "first," "second," "third," "fourth," and the like in the description and in the claims of the present application and in the above-described drawings (if any) are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be implemented in other sequences than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

At present, a data transmission task program is seriously coupled with a specific service, and the universal capability is lacked in the aspect of heterogeneous database synchronization, so that when the service requirement is newly increased, the expansibility and the reusability are both problematic. Since the data transfer is coupled with the written code, any part of which changes, the transfer program needs to be rewritten, and the code reuse degree is low. For example, if a new requirement for transmitting data from MongoDB to MySQL or from Oracle to MySQL comes, a new program needs to be developed to realize the existing program for transmitting data from MySQL to MySQL.

Resulting in inefficient data transfer between disparate databases.

Therefore, in order to solve the above problem, an embodiment of the present application provides a data transmission method, which divides a data transmission task into three parts for a transmission requirement of a DTS: the reading part, the format conversion part and the writing part provide a DTS service product with general capability, namely, the upper and lower sources are decoupled, and the support capability of the upstream and downstream types can be rapidly expanded. For the sake of understanding, the method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

Referring to fig. 1, as shown in fig. 1, an embodiment of a data transmission method according to the present application includes the following steps.

101. And acquiring the first plug-in according to the data type of the first database.

In this embodiment, the first plug-in is configured to read first data in a first database, where the first plug-in may be a section of code running in a server, and the server obtains the first plug-in by calling the section of code, and the first database is a database from which data is derived.

It should be noted that, the method provided in the embodiment of the present application may be executed by a server or a terminal, and the embodiment of the present application is not limited thereto.

102. And acquiring the second plug-in according to the data type of the second database.

In this embodiment, the second plugin is used to write second data into the second database, the second database and the first database are different types of databases, that is, the first database and the second database are heterogeneous databases, the second data and the first data are different types of data, the second plugin may be a section of code running in the server, the server obtains the second plugin by calling the section of code, and the second database is a database into which data is written.

103. And acquiring a third plug-in according to the data type of the first database and the data type of the second database.

In this embodiment, the third plugin is configured to convert the first data read by the first plugin into second data and send the second data to the second plugin. Therefore, the second plug-in can write the second data after being converted into the format into the second database, so that data transmission serving as a heterogeneous database between the first database and the second database is realized, optionally, the third plug-in can be a section of code running in the server, and the server can acquire the third plug-in by calling the section of code.

In this embodiment, the data transmission task is divided into three plug-ins in a decoupling manner, where a first plug-in is a read plug-in, a second plug-in is a write plug-in, and a third plug-in is a format conversion plug-in. For example, if there are three read plug-ins A, B, and C and three write plug-ins D, E, and F, no consideration is given to the change of the format conversion plug-ins, then theoretically, 9 different data transfer tasks in total, A to D, A to E, A to F, B to D, \ 8230, and C to F, can be supported, and the supported transfer task types are multiplied if only one read plug-in or write plug-in is newly implemented. By the method, when the data transmission task needs to change the type of the data source (namely, the first database) or the data destination (namely, the second database) due to the change of the requirement, the data transmission task can be realized only by developing the corresponding reading plug-in or writing plug-in (which can be directly used if the corresponding plug-in exists), so that the code development amount is reduced, and the data transmission efficiency is improved.

It should be noted that, in a specific operation, the server initiates data transmission of the heterogeneous databases based on a request of the terminal, and when the terminal sends a request message to the server, the terminal attaches a configuration file, where content recorded in the configuration file is used to instruct the server to perform data transmission, for example, the configuration file includes identifiers of the first database and the second database, so that the server knows that the data transmission task of this time is to transmit data of the first database to the second database, and further, the configuration file also includes data types of the first database and the second database, so that the server can obtain corresponding plug-ins according to the configuration types in the configuration file, and for convenience of understanding, the following description will be given in detail with reference to the accompanying drawings.

Referring to fig. 2, as shown in fig. 2, an embodiment of a data transmission method provided in the present application includes the following steps.

201. And acquiring a configuration file.

In this embodiment, the configuration file may be generated and sent by the terminal to the server, or may be generated by the terminal itself and executed by the terminal, which is not limited in this embodiment of the present application, and the record information in the configuration file at least includes data types of the first database and the second database, and optionally, may further include address information of the databases and port information of each plug-in, and the like.

202. And acquiring the first plug-in according to the data type of the first database recorded in the configuration file.

In this embodiment, the server is preset with reading plug-ins corresponding to different types of databases, so that the reading plug-in corresponding to the data source can be obtained according to the data type of the data source (i.e., the data type of the first database) recorded in the configuration file.

203. And acquiring the second plug-in according to the data type of the second database recorded in the configuration file.

In this embodiment, the server is preset with write plug-ins corresponding to different types of databases, so that the write plug-in corresponding to the data destination can be obtained according to the data type of the data destination (i.e., the data type of the second database) recorded in the configuration file.

204. And acquiring a third plug-in according to the data type of the first database and the data type of the second database recorded in the configuration file.

In this embodiment, according to the data types of the data source and the data destination (that is, the data type of the first database and the data of the second database) recorded in the configuration file, a preset format conversion plug-in is obtained for converting the data format of the data source into the data format of the data destination.

It should be noted that, in the system architecture of the DTS system, as shown in fig. 3, the terminal 301 may send a task request to the interaction unit 3021 of the server 302, where the task request may be a create task, a terminate task, or an query task, where the task request includes a configuration file, that is, the step 201 is completed, then the interaction unit 3021 delegates the work engine 3022 of the server to obtain a corresponding plug-in, the work engine 3022 executes the steps 202 to 203 according to the plug-in, obtains a read plug-in, a format conversion plug-in, and a write plug-in, then the work engine 3022 configures the three plug-ins into a work process 3023, and the work process, during execution, reads data from the data source 303 and writes the data into the data destination 304, thereby completing data transmission between heterogeneous databases.

Further, referring to fig. 4, fig. 4 illustrates a process from task establishment to task execution of the data transmission method according to the embodiment of the present application.

As shown in fig. 4, the work engine 401 configures and acquires a work process 402 (i.e., the work process 3023) as described above, where the work process 402 includes a read plug-in, a format conversion plug-in, and a write plug-in acquired by the work engine 401 according to the configuration file, and then the work engine 401 adds the work process 402 to the wait queue 403, and if the wait queue is not empty and there is a currently idle work thread, the work thread will take one work process 402 out of the queue to start execution, and in the execution process, acquire data from the data source 404, and write the data into the data destination 405 after sequentially processing by the read plug-in 4021, the format conversion plug-in 4022, and the write plug-in 4023. Therefore, the data transmission method provided by the embodiment of the application is realized.

The first embodiment and the second embodiment describe in detail the plug-in obtaining method in the data transmission method provided in the embodiments of the present application. The following describes the specific use of each insert in the actual operation.

Referring to fig. 5, as shown in fig. 5, a third embodiment of the data transmission method provided in the present application includes the following steps.

501. A data transfer service, DTS, request is obtained.

In this embodiment, a specific implementation manner of this step may be that described in fig. 3, the terminal 301 sends a task request to the interaction unit 3021 of the server 302, where the task request includes a data transfer service DTS request, so that the server may trigger execution of a subsequent step based on the data transfer service DTS request.

502. And starting to acquire the first plug-in according to the DTS request.

In this embodiment, the DTS request triggers the server to execute the data transmission method provided in this embodiment, so that the server can obtain the first plug-in through the method described in

step

101 or 202.

503. And converting the first data into the data in the universal format through the first plug-in.

In this embodiment, the common format data is a format that can be read by the second plug-in and the third plug-in. Alternatively, step 503 may be specifically realized by the following steps.

A first thread is started by a first plug-in, the first thread for exporting first data from a first database.

In this embodiment, the first database is read by the first thread, so as to derive data of the data source.

And converting the first data into the data with the common format and then putting the data into a first queue.

In this embodiment, the data read from the data source is converted into the data in the general format and then placed into the first queue, so as to facilitate reading in the subsequent steps.

A second thread is started by the first plug-in.

In this embodiment, the first plug-in starts the second thread, the second thread is independent from the first thread, and the two threads work independently without interference.

The common format data is called from the first queue by the second thread.

In this embodiment, the second thread created by the first plug-in obtains the common format data from the first queue. Therefore, the first plug-in realizes the reading of data source data and the conversion of data format through two threads respectively.

504. And sending the common format data to the third plug-in through the first plug-in.

In this embodiment, the first plug-in sends the common format data to the third plug-in, so that the third plug-in executes the subsequent steps.

505. And converting the common format data into second data through a third plug-in.

In this embodiment, the data in the common format is of a data type that can be read by the third plug-in, so that the third plug-in converts the data in the common format into second data that can be written into the second database after reading the data in the common format, thereby implementing the conversion of the data format.

506. And sending the second data to the second plug-in through the third plug-in.

In this embodiment, the third plugin sends the second data to the second plugin, so that the second plugin performs the subsequent steps. The second data is the first data after format conversion.

507. And writing the second data into the second database through the second plug-in.

In this embodiment, after the second plug-in obtains the second data, the second data is written into the second database, so that data transmission between heterogeneous databases is realized.

In this embodiment, in combination with a specific use scenario of the DTS, the read plug-in, the format conversion plug-in, and the write plug-in are called according to the conditions of the data source and the data destination, so that data transmission of the data transmission method provided by the embodiment of the present application to the heterogeneous database is achieved.

In the data transmission method provided by the present application, the obtaining method and the using method of the plug-in are introduced, and compared with the method of coupling the data transmission code to the transmission task program in the prior art, the data transmission task is decoupled from the task program by setting an independent plug-in, so that a more flexible configuration method is realized.

Specifically, for scenes of data synchronization through the DTS for a long time, such as online analysis and real-time disaster recovery, it is required that the DTS transmits strict information to ensure that the information is not lost. However, due to system upgrade, process crash, machine failure, network failure, etc., these operations or failures are likely to cause the DTS transmission module to fail.

In order to ensure the reliability of transmission, the current technology mainly adopts the following scheme.

The data stream transmitted during data transmission can be abstracted into a series of messages generated in an upstream sequence, wherein the upstream refers to a reading side for reading a data source, each message corresponds to a unique offset position, and the offset position is specifically an offset of a message file relative to the beginning of the file when the message file is stored on a disk. When a DTS transmission task is started, a certain offset position at the upstream is designated, all messages after the position are transmitted, and the upstream offset position is updated in the transmission process at regular time so as to be transmitted continuously at the next starting time. It is ensured that data is not lost when the task is restarted, and it is actually ensured that the offset position recorded during the running of the task cannot be ahead of the offset position corresponding to the message actually transmitted to the downstream.

However, in the data transmission method provided by the embodiment of the present application, the upstream and downstream implement a decoupled system in a plug-in form, and the upstream read plug-in does not know whether the downstream write plug-in writes a message to the destination database; the downstream write-in plug-in does not know the upstream offset position corresponding to the message which is successfully transmitted by the downstream write-in plug-in, so the plug-in cannot control the storage of the offset progress. Second, for parallel transfer mode, it is possible that messages with offset positions later are written downstream first, while messages with offset positions earlier are not delivered.

Therefore, to solve the above problems, embodiments of the present application provide a specific solution, and for easy understanding, the following detailed description is made with reference to the accompanying drawings.

Referring to fig. 6, as shown in fig. 6, based on the plug-in that has been acquired in the above steps, in the process of ensuring the transmission reliability, a fourth embodiment of the data transmission method provided in the embodiment of the present application includes the following steps.

601. The first version number is marked on the first data acquired by the first plug-in.

In this embodiment, the first version number is a globally unique version number, and after the first plug-in reads data from the first database (i.e., the data source), the first version number is marked on the read first data, so that the first version number can uniquely mark the first data, and the first data is distinguished from other data. In a specific implementation process, the data granularity of the first data may be defined by a user, and is not limited thereto.

602. The first version number is sent to a progress controller.

In this embodiment, the schedule controller is configured to record a corresponding relationship between the first data and the first version number, and optionally, the schedule controller may be a multi-version concurrent control (MVCC) schedule controller.

603. And after the first data is converted into second data by the third plug-in and is written into the second database by the second plug-in, recording the data corresponding to the first version number as transmitted in the progress controller.

In this embodiment, the schedule controller is configured to record data to be written into a second database (data destination), and record the version number as transmitted, so that it can be known that the data corresponding to the version number has been written into the second database.

604. And when the second plug-in writes second data into the second database, acquiring a second version number corresponding to the second data from the progress controller.

605. If the second version number is recorded as transmitted in the progress controller, the second plug-in does not write the second data to the second database.

In this embodiment, the second version number is a globally unique version number, and when breakpoint retransmission occurs, the second plugin acquires the version number corresponding to the currently transmitted data from the progress controller before writing the data, that is, the second plugin can accurately transmit the message that has not been transmitted, for example, the second plugin is restarted after being suspended, and currently needs to write the second data into the second database, and first queries in the progress controller, and whether the second version number corresponding to the second data is marked as committed, if the second version number is committed, it indicates that the second data has been written before the transmission is suspended, and therefore the second data is not repeatedly transmitted, so that repeated writing of the data due to breakpoint retransmission is avoided, and the reliability of transmission is improved.

In this embodiment, the read and written data are recorded by setting the progress controller, and after the write plug-in determines that a message is successfully written into the data destination, the progress controller only needs to notify that the version number of the message is set as committed. The progress controller will periodically scan and find the progress corresponding to the message with the latest version number in all submitted messages, which is also called low-level progress, and store the progress into the task meta-information storage. Since the low water level schedule strictly ensures that all previous message versions have been committed, restarting from the low water level schedule when a task breakpoint resumes does not result in skipping any messages that have not yet been sent.

Optionally, in order to reduce memory usage of the controller, the offset information of the message may be divided into two parts, namely static and dynamic parts, and the progress controller only stores one part of the static information and stores multiple versions of the dynamic offset information. Namely: storing the data recorded as transmitted in the progress controller; and storing the data which is not recorded as transmitted in the progress controller in a cache.

It should be noted that, in order to ensure the reliability of data transmission, when an upstream write plug-in transmits a message downstream, three return states may be encountered, i.e., success, failure, or timeout. For messages that return success/failure, it is explicitly known that no retransmission of the message is needed/required at this time. However, for the message returning overtime, it may be lost in the process of sending DTS to the downstream, that is, the downstream does not actually receive the message and needs to resend the message; it is also possible that ACK is lost when the message has been received downstream and ACK is returned to the DTS without retransmission. This problem is also known as the two general problem.

In order to solve this problem, embodiments of the present application further provide an implementation manner to implement an idempotent transmission design in a data transmission process. For a database system, and a distributed file system or a message queue system, there are different implementations, and for easy understanding, the following detailed description is made with reference to the solutions under different systems in the drawings.

1. A database system.

Referring to fig. 7, as shown in fig. 7, based on the plug-in that has been acquired in the above steps, in the process of ensuring the transmission reliability, a fifth embodiment of the data transmission method provided in the embodiment of the present application includes the following steps.

701. And acquiring a database main key or a unique key of the first data through the first plug-in.

In this embodiment, in the database system, after the first plug-in reads the first data in the first database, the primary key or the unique key of the first data is obtained, where the primary key of the database refers to a combination of one or more columns, and its value can uniquely identify each row in the table, through which the physical integrity of the table can be enforced. The main key is mainly used for being associated with the external keys of other tables and modifying and deleting the record.

702. And converting the first data into second data through the third plug-in.

In this embodiment, the format conversion plug-in and the third plug-in keep the corresponding relationship between the primary key or the unique key and the data in the process of converting the first data into the second data.

703. When the second data is written into the second database through the second plugin, if the primary key or the unique key corresponding to the second database is written into the second database, the second plugin does not write the second data any more.

In this embodiment, it can be determined through the primary key or the unique key whether the second data to be written currently has already been written in the second database system, thereby avoiding repeated writing of data.

In the embodiment, the main key or the unique key of the data is identified according to the actual working condition of the database system, so that each data is ensured to be written once, repeated input writing is avoided, and idempotent of data transmission is realized in the database system.

2. A distributed file system or a message queue system.

Referring to fig. 8, as shown in fig. 8, based on the plug-in that has been acquired in the above steps, in the process of ensuring the transmission reliability, a sixth embodiment of the data transmission method provided in the embodiment of the present application includes the following steps.

801. The first data is accompanied by first offset position information of the first data by the first plug-in.

In this embodiment, after the first plug-in reads the first data in the first database, the first plug-in attaches first offset position information of the first data to the first data, the first offset position information being used to record an offset from the beginning of the file when the first data is stored on the disk. So that it can be positioned to the offset position of the first data by the first offset information.

802. And converting the first data into second data through the third plug-in.

In this embodiment, the first data is converted into the second data by the third plug-in, where a corresponding relationship between the second data and the first offset position is retained.

803. And acquiring first offset position information of at least one second data through a second plug-in, and rejecting second data corresponding to repeated first offset information when the repeated first offset information exists.

In this embodiment, in the transmission process, retransmission may occur due to transmission errors, and a situation that the second plug-in (i.e., the write plug-in) receives multiple duplicate messages may occur, so that the duplicate messages may be identified by the offset position information attached to the messages, and the duplicate messages are proposed.

In this embodiment, the data is identified by adding the offset position information to the data, so that the read plug-in does not need to identify whether the data is repeated, and the task of removing the duplicate is directly transferred to the downstream write plug-in. Thereby achieving idempotency of the downstream system.

In the fifth embodiment and the sixth embodiment, a scheme for improving the transmission reliability in the data transmission method provided in the embodiments of the present application is described in detail. In the actual working process, a transmission fault still possibly occurs, and in order to ensure the availability of data transmission, when the data transmission fails, measures need to be taken to recover the failure, so that the embodiment of the application further provides a scheme for improving the availability of the decoupled plug-in system.

Referring to fig. 9, as shown in fig. 9, a seventh embodiment of the data transmission method according to the present application includes the following steps.

901. And when detecting that the reading of the first plug-in unit has an error or detecting that the writing of the second plug-in unit has an error, restarting the first plug-in unit and the second plug-in unit.

In this embodiment, the read error may be a case where reading is impossible or a case where reading is abnormal, and the embodiment of the present application is not limited thereto, and the write error may be a case where writing is impossible or a case where writing is abnormal. Further, the exception may also include transmission exception of other links of data transmission, and at this time, the exception is attempted to be solved by reconnection itself through restart.

902. And when the first plug-in and the second plug-in still can not work normally after being restarted, generating an error code.

In this embodiment, when the reconnection attempt fails or the determination fails to solve the problem through reconnection, the error condition is thrown out in the form of an error code.

903. And acquiring a preset recovery strategy according to the error code.

In this embodiment, the management and control layer in the server obtains the error code generated in step 902, and the management and control layer further analyzes the error, and since the corresponding relationship between the error code and the recovery policy is a preset relationship, the management and control layer can obtain the preset recovery policy according to the error code.

904. And recovering the transmission of the first plug-in and the second plug-in according to the recovery strategy.

In this embodiment, according to the obtained recovery strategy, the transmission of the plug-in is recovered, that is, the self-healing of the data transmission system can be realized, so that the recovery of the fault is realized.

It should be noted that, for the above recovery policy, different recovery policies are triggered according to different error conditions to implement self-healing of the data transmission system, so the embodiment of the present application does not limit details, and as an example, the embodiment of the present application provides a scheme for implementing the recovery policy when a master-slave handover occurs in an upstream database.

Referring to fig. 10, as shown in fig. 10, the method specifically includes the following steps.

1001. When the first database is subjected to master-slave switching, first type information of the first database after the master-slave switching is acquired, wherein the first type information is used for recording the type of the first database.

In this embodiment, when the main database of the database is down, data reading cannot be performed any more, so that the data source actively performs master-slave switching to be switched to the standby database, and at this time, because the data source is changed, the server needs to acquire a new reading plug-in again to read a new data source.

1002. A first plug-in corresponding to the first type of information is obtained.

In this embodiment, after the type information of the new data source is obtained, the first plug-in corresponding to the data source is obtained according to the type information, so that the new plug-in can continue to read the data of the data source to perform the operation of the subsequent step, thereby implementing the self-healing of the transmission system.

The following steps may refer to any of the above embodiments, and are not described herein again.

The above description is directed to the solutions provided in the embodiments of the present application. It will be appreciated that the computer device, in order to implement the above-described functions, comprises corresponding hardware structures and/or software modules for performing the respective functions. Those of skill in the art will readily appreciate that the various illustrative modules and algorithm steps described in connection with the embodiments disclosed herein may be implemented as hardware or combinations of hardware and computer software. Whether a function is performed as hardware or computer software drives hardware depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

Described in terms of hardware structures, the method may be implemented by one entity device, may also be implemented by multiple entity devices together, and may also be a logic function module in one entity device, which is not specifically limited in this embodiment of the present application.

For example, the above methods may all be implemented by the computer device in fig. 11. Fig. 11 is a schematic hardware structure diagram of a computer device according to an embodiment of the present application. The computer device includes at least one processor 1101, communication lines 1102, memory 1103, and at least one communication interface 1104.

The processor 1101 may be a general processing unit (CPU), a microprocessor, an application-specific integrated circuit (server IC), or one or more ICs for controlling the execution of programs in accordance with the present invention.

The communication link 1102 may include a path for communicating information between the aforementioned components.

Communication interface 1104, using any transceiver or like device for communicating with other devices or communication networks, such as ethernet, radio Access Network (RAN), wireless Local Area Networks (WLAN), etc.

The memory 1103 may be, but is not limited to, a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a Random Access Memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disk read-only memory (CD-ROM) or other optical disk storage, optical disk storage (including compact disk, laser disk, optical disk, digital versatile disk, blu-ray disk, etc.), magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. The memory may be separate and coupled to the processor via a communication line 1102. The memory may also be integrated with the processor.

The memory 1103 is used for storing computer-executable instructions for executing the present invention, and is controlled by the processor 1101. The processor 1101 is configured to execute computer-executable instructions stored in the memory 1103, thereby implementing the methods provided by the above-described embodiments of the present application.

Optionally, the computer-executable instructions in the embodiments of the present application may also be referred to as application program codes, which are not specifically limited in the embodiments of the present application.

In particular implementations, processor 1101 may include one or more CPUs, such as CPU0 and CPU1 in fig. 11, as one embodiment.

In particular implementations, a computer device may include multiple processors, such as processor 1101 and processor 1107 in FIG. 11, as an embodiment. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, and/or processing cores for processing data (e.g., computer program instructions).

In particular implementations, a computer device may also include an output device 1105 and an input device 1106, as one embodiment. The output device 1105 is in communication with the processor 1101 and may display information in a variety of ways. For example, the output device 1105 may be a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display device, a Cathode Ray Tube (CRT) display device, a projector (projector), or the like. The input device 1106 is in communication with the processor 1101 and may receive user input in a variety of ways. For example, the input device 1106 may be a mouse, keyboard, touch screen device, or sensing device, among others.

The computer device may be a general purpose device or a special purpose device. In a specific implementation, the computer device may be a desktop computer, a laptop computer, a network server, a Personal Digital Assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, an embedded device, or a device with a similar structure as in fig. 11. The embodiment of the application does not limit the type of the computer equipment.

In the embodiment of the present application, the storage device may be divided into the functional units according to the above method example, for example, each functional unit may be divided corresponding to each function, or two or more functions may be integrated into one processing unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit. It should be noted that the division of the unit in the embodiment of the present application is schematic, and is only a logic function division, and there may be another division manner in actual implementation.

For example, in the case where the respective functional units are divided in an integrated manner, fig. 12 shows a schematic diagram of a data transmission apparatus.

As shown in fig. 12, the data transmission apparatus provided in the embodiment of the present application includes:

a first obtaining unit 1201, where the first obtaining unit 1201 is configured to obtain a first plug-in according to a data type of a first database, where the first plug-in is configured to read first data in the first database;

a second obtaining unit 1202, where the second obtaining unit 1202 is configured to obtain a second plugin according to a data type of a second database, where the second plugin is configured to write second data into the second database, the second database and the first database are different types of databases, and the second data and the first data are different types of data;

a third obtaining unit 1203, where the third obtaining unit 1203 is configured to obtain a third plugin according to the data type of the first database and the data type of the second database, and the third plugin is configured to convert the first data read by the first plugin, which is obtained by the first obtaining unit 1201, into the second data and send the second data to the second plugin, which is obtained by the second obtaining unit 1202.

Optionally, the apparatus further comprises:

a fourth obtaining unit 1204, where the fourth obtaining unit 1204 is configured to obtain a configuration file, where data types of the first database and the second database are recorded in the configuration file;

the first obtaining unit 1201 is further configured to:

acquiring the first plugin according to the data type of the first database recorded in the configuration file acquired by the fourth acquiring unit 1204;

the second obtaining unit 1202 is further configured to:

acquiring the second plugin according to the data type of the second database recorded in the configuration file acquired by the fourth acquiring unit 1204;

the third obtaining unit 1203 is further configured to:

the third plug-in is obtained according to the data type of the first database and the data type of the second database recorded in the configuration file obtained by the fourth obtaining unit 1204.

Optionally, the apparatus further comprises:

a first converting unit 1205, where the first converting unit 1205 is configured to convert the first data into common format data through the first plug-in, and the common format data is a format readable by the second plug-in and the third plug-in;

a second conversion unit 1206, wherein the second conversion unit 1206 is configured to convert the common format data converted by the first conversion unit 1205 into the second data through the third plug-in;

a writing unit 1207, the writing unit 1207 is configured to write the second data transformed by the second transforming unit 1206 into the second database through the second plugin.

Optionally, the apparatus further includes a fifth obtaining unit 1208, where the fifth obtaining unit 1208 is configured to: acquiring a Data Transmission Service (DTS) request;

the first obtaining unit 1201 is further configured to:

the acquiring of the first plug-in is started according to the DTS request acquired by the fifth acquiring unit 1208.

Optionally, the first converting unit 1205 is further configured to:

converting the first data into the universal format data and then putting the universal format data into a first queue;

starting a second thread through the first plug-in;

calling the common format data from the first queue through the second thread;

Optionally, the apparatus further includes a first executing unit 1209, where the first executing unit 1209 is configured to:

Optionally, the first execution unit 1209 is further configured to:

storing the data recorded as transmitted in the progress controller;

Optionally, the first database and the second database are both database systems, the apparatus further includes a second executing unit 1210, and the second executing unit 1210 is configured to:

Optionally, the first database and the second database are both distributed file systems or message queue systems, and the apparatus further includes a third executing unit 1211, where the third executing unit 1211 is configured to:

Optionally, the apparatus further includes a fourth executing unit 1212, where the fourth executing unit 1212 is configured to: and restarting the first plug-in and the second plug-in when detecting that the reading of the first plug-in is wrong or detecting that the writing of the second plug-in is wrong.

Optionally, the fourth executing unit 1212 is further configured to:

Optionally, the apparatus further comprises a fifth execution unit 1213, the fifth execution unit 1213 is configured to: when the first database is subjected to master-slave switching, first type information of the first database after the master-slave switching is acquired, wherein the first type information is used for recording the type of the first database;

a first plug-in corresponding to the first type information is obtained.

Further, an embodiment of the present invention also provides a computer storage medium, which includes instructions that, when executed on a computer device, cause the computer device to execute the above method.

For a detailed description of the program stored in the computer storage medium provided in the embodiments of the present application, reference may be made to the above embodiments, which are not described herein again.

The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.

Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of data transmission, comprising:

acquiring a second plugin according to the data type of a second database, wherein the second plugin is used for writing second data into the second database, the second database and the first database are different types of databases, and the second data and the first data are different types of data;

acquiring a third plugin according to the data type of the first database and the data type of the second database, wherein the third plugin is used for converting the first data read by the first plugin into the second data and sending the second data to the second plugin;

marking a first version number for the first data acquired by the first plug-in, wherein the first version number is a globally unique version number;

2. The method of claim 1, further comprising:

acquiring a configuration file, wherein the data types of the first database and the second database are recorded in the configuration file;

the obtaining of the first plug-in according to the data type of the first database includes:

acquiring the first plug-in according to the data type of the first database recorded in the configuration file;

the obtaining of the second plug-in according to the data type of the second database includes:

acquiring the second plug-in according to the data type of the second database recorded in the configuration file;

the obtaining a third plugin according to the data type of the first database and the data type of the second database includes:

and acquiring the third plug-in according to the data type of the first database and the data type of the second database recorded in the configuration file.

3. The method of claim 1, wherein after obtaining the first plug-in according to the data type of the first database, further comprising:

converting the first data into common format data through the first plug-in, wherein the common format data is a format which can be read by the second plug-in and the third plug-in;

converting the common format data into the second data through the third plug-in;

writing, by the second plug-in, the second data to the second database.

4. The method of claim 3, wherein prior to obtaining the first plug-in according to the data type of the first database, further comprising:

acquiring a Data Transmission Service (DTS) request;

and starting to acquire the first plug-in according to the DTS request.

5. The method of claim 4, wherein said converting said first data into common format data by said first plug-in comprises:

starting a second thread through the first plug-in;

calling the common format data from the first queue through the second thread;

after the first data is converted into the data in the common format by the first plug-in, the method further comprises the following steps:

and sending the common format data to the third plug-in through the first plug-in.

6. The method of claim 1, further comprising:

storing the data recorded as transmitted in the progress controller;

7. The method of any one of claims 1 to 5, wherein the first database and the second database are database systems, the method further comprising:

when second data is written into the second database through the second plugin, if the primary key or the unique key corresponding to the second database is already written into the second database, the second plugin does not write the second data any more.

8. The method of any of claims 1 to 5, wherein the first database and the second database are both distributed file systems or message queue systems, the method further comprising:

attaching first offset position information of the first data to the first data by the first plug-in, the first offset position information being used to record an offset from a file start when the first data is stored on a disk;

9. The method of any of claims 1 to 5, wherein the first plugin and the second plugin are restarted when a read error of the first plugin or a write error of the second plugin is detected.

10. The method of claim 9, further comprising:

when the first plugin and the second plugin still cannot work normally after being restarted, generating an error code;

11. The method of claim 10, wherein the recovery policy comprises: when the first database is subjected to master-slave switching, first type information of the first database after the master-slave switching is acquired, wherein the first type information is used for recording the type of the first database;

and acquiring a first plug-in corresponding to the first type information.

12. A data transmission apparatus, comprising:

a third obtaining unit, configured to obtain a third plugin according to a data type of the first database and a data type of the second database, where the third plugin is configured to convert the first data read by the first plugin obtained by the first obtaining unit into the second data and send the second data to the second plugin obtained by the second obtaining unit;

a first execution unit, configured to mark a first version number for the first data acquired by the first plugin, where the first version number is a globally unique version number; sending the first version number to a progress controller, wherein the progress controller is used for recording the corresponding relation between the first data and the first version number; when the first data is converted into the second data by the third plug-in and is written into the second database by the second plug-in, recording the data corresponding to the first version number as transmitted in the progress controller; when the second plug-in writes the second data into the second database, acquiring a second version number corresponding to the second data from the progress controller, wherein the second version number is a globally unique version number; if the second version number is recorded as transmitted in the progress controller, the second plug-in does not write the second data to the second database.

13. A computer device, characterized in that the computer device comprises: an interaction device, an input/output (I/O) interface, a processor, and a memory having program instructions stored therein;

the interaction device is used for acquiring an operation instruction input by a user;

the processor is configured to execute program instructions stored in the memory to perform the method of any of claims 1-11.

14. A computer-readable storage medium comprising instructions that, when executed on a computer device, cause the computer device to perform the method of any one of claims 1-11.