WO2021179589A1

WO2021179589A1 - Data transmission method and data channel system

Info

Publication number: WO2021179589A1
Application number: PCT/CN2020/119709
Authority: WO
Inventors: 任超; 李大虎
Original assignee: 北京旷视科技有限公司
Priority date: 2020-03-13
Filing date: 2020-09-30
Publication date: 2021-09-16
Also published as: CN111405035A

Abstract

The present application relates to a data transmission method and a data channel system. The method is applied to a data channel system. The method comprises: a gateway control module receives first data in a first format sent by a database, and sends the first data to a data channel cluster; the data channel cluster converts the first format of the first data into a second format to obtain first data subjected to format conversion, and sends the first data subjected to format conversion to an application system; the data channel cluster receives second data in a third format sent by the application system, converts the third format of the second data into a fourth format to obtain second data subjected to format conversion, and sends the second data subjected to format conversion to the gateway control module; the gateway control module sends the second data subjected to format conversion to a database. Use of the present application can achieve data interaction between an application system and a database.

Description

Data transmission method and data channel system

This application is based on a Chinese patent application with an application number of 202010174731.3 and an application date of March 13, 2020, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated by reference into this application.

Technical field

This application relates to the field of computer technology, in particular to a data transmission method and a data channel system.

Background technique

At present, video image information application systems (hereinafter referred to as application systems) have been widely used, and there are more and more types. However, the application system usually does not have the function of sending and receiving data, which leads to the inability to transmit data between different application systems, which in turn leads to the inability to coordinate the video image application between the users of the application systems.

In order to solve the above problems, the application system data standard specification was formulated. In the standard specification, the video image information database (hereinafter referred to as the database) is used as the hub of data interaction, and each application system realizes data interaction through the database.

However, the application system mainly focuses on the realization of video image application functions in the early stage of development, and does not support standard specifications and databases. Therefore, when the application system needs to interact with the database for data, it is necessary to spend manpower and material resources on the secondary development of the application system.

Summary of the invention

Based on this, it is necessary to provide a data transmission method and a data channel system in response to the above technical problems.

In a first aspect, a data transmission method is provided, the method is applied to a data channel system, the data channel system includes a gateway control module and a data channel cluster, the gateway control module is connected to a database, and the data channel cluster Connecting to the application system, the method includes:

The gateway control module receives the first data in the first format sent by the database, and sends the first data to the data channel cluster; the first format is that the first data is stored in the database Format

The data channel cluster converts the first format of the first data into a second format, obtains the first data after the format conversion, and sends the first data after the format conversion to the application system; The second format is a format in which the first data is stored in the application system;

The data channel cluster receives the second data in the third format sent by the application system, converts the third format of the second data into a fourth format, obtains the second data after the format conversion, and combines The second data after the format conversion is sent to the gateway control module; the third format is the format for storing the second data in the application system, and the fourth format is for storing the first data in the database. 2. The format of the data;

The gateway control module sends the second data after the format conversion to the database.

As an optional implementation manner, the gateway control module includes an application program interface unit, the application program interface unit includes an application program interface corresponding to the first data, and the data channel cluster includes an application program interface corresponding to the first data. The first data channel;

The gateway control module receiving the first data in the first format sent by the database and sending the first data to the data channel cluster includes:

The application program interface unit receives the first data in the first format sent by the database through the first application program interface, and sends the first data to the first data channel in the data channel cluster .

As an optional implementation manner, the data channel cluster includes a first data channel corresponding to the first data, and the first data channel includes a first data mapping table and first data corresponding to the first data. driver;

The data channel cluster converts the first format of the first data into a second format, obtains the first data after the format conversion, and sends the first data after the format conversion to the application system, include:

Converting, by the first data channel, the first format of the first data into a second format according to the first data mapping table, to obtain the first data after the format conversion;

The first data channel sends the first data after the format conversion to the application system through the first data driver.

As an optional implementation manner, the data channel cluster includes a second data channel corresponding to the second data, and the second data channel includes a second data mapping table and second data corresponding to the second data. driver;

The data channel cluster receives the second data in the third format sent by the application system, converts the third format of the second data into a fourth format, obtains the second data after the format conversion, and combines The second data after the format conversion is sent to the gateway control module, including:

The second data channel receives the second data in the third format sent by the application system through the second data driver;

The second data channel converts the third format of the second data into a fourth format according to the second data mapping table to obtain the second data after the format conversion, and the second data after the format conversion is obtained. The second data is sent to the gateway control module.

As an optional implementation manner, the gateway control module includes a gateway unit;

The gateway control module sending the second data after the format conversion to the database includes:

The gateway control module sends the second data after the format conversion to the database through the gateway unit.

As an optional implementation manner, the data channel system further includes a channel management module, and the method further includes:

The channel management module obtains the data driver and the data mapping table corresponding to each data;

The channel management module creates a data channel corresponding to each data according to a data driver and a data mapping table corresponding to each data to form the data channel cluster.

As an optional implementation manner, the data channel system includes multiple data channel clusters, and the multiple data channel clusters include a primary data channel cluster and a backup data channel cluster, and the method further includes:

When the channel management module detects the failure of the first data channel cluster, the second data channel cluster is switched to the main data channel cluster, the first data channel cluster is the main data channel cluster, and the second data channel cluster is Backup data channel cluster.

As an optional implementation manner, the data channel system further includes a metadata management module, and the method further includes:

The metadata management module obtains metadata of each data;

The metadata management module analyzes and processes the metadata of each data to obtain data driver metadata and data mapping table metadata corresponding to each data;

The metadata management module performs translation processing on the data driver metadata and data mapping table metadata corresponding to each data to obtain the data driver and data mapping table corresponding to each data, and stores the data corresponding to each data Drive and data mapping table.

In a second aspect, a data channel system is provided, the data channel system includes a gateway control module and a data channel cluster, the gateway control module is connected to a database, and the data channel cluster is connected to an application system;

The gateway control module is configured to receive first data in a first format sent by the database, and send the first data to the data channel cluster; the first format is that the database stores the The format of the first data;

The data channel cluster is configured to convert the first format of the first data into a second format, obtain the first data after the format conversion, and send the first data after the format conversion to the Application system; the second format is a format in which the first data is stored in the application system;

The data channel cluster is configured to receive second data in a third format sent by the application system, convert the third format of the second data into a fourth format, and obtain the second data after the format conversion, And send the second data after the format conversion to the gateway control module; the third format is the format in which the second data is stored in the application system, and the fourth format is the format stored in the database The format of the second data;

The gateway control module is configured to send the second data after the format conversion to the database.

The application program interface unit is configured to receive first data in a first format sent by the database through the first application program interface, and send the first data to the first data in the data channel cluster One data channel.

The first data channel is configured to convert the first format of the first data into a second format according to the first data mapping table to obtain the first data after the format conversion;

The first data channel is used to send the first data after the format conversion to the application system through the first data driver.

The second data channel is configured to receive the second data in the third format sent by the application system through the second data driver;

The second data channel is configured to convert the third format of the second data into a fourth format according to the second data mapping table to obtain the second data after the format conversion, and convert the format The latter second data is sent to the gateway control module.

The gateway control module is configured to send the second data after the format conversion to the database through the gateway unit.

As an optional implementation manner, the data channel system further includes a channel management module;

The channel management module is used to obtain the data driver and the data mapping table corresponding to each data;

The channel management module is configured to create a data channel corresponding to each data according to a data driver and a data mapping table corresponding to each data to form the data channel cluster.

As an optional implementation manner, the data channel system includes multiple data channel clusters, and the multiple data channel clusters include a primary data channel cluster and a backup data channel cluster;

The channel management module is configured to switch the second data channel cluster to the main data channel cluster when the failure of the first data channel cluster is detected, the first data channel cluster is the main data channel cluster, and the second data channel cluster is the main data channel cluster. The channel cluster is a backup data channel cluster.

As an optional implementation manner, the data channel system further includes a metadata management module;

The metadata management module is used to obtain metadata of each data;

The metadata management module is configured to analyze and process the metadata of each data to obtain data driver metadata and data mapping table metadata corresponding to each data;

The metadata management module is used to translate the data driver metadata and data mapping table metadata corresponding to each data to obtain the data driver and data mapping table corresponding to each data, and to store the data Corresponding data driver and data mapping table.

In a third aspect, a computer device is provided, including a memory and a processor, the memory stores a computer program that can run on the processor, and when the processor executes the computer program, any one of the first aspect is implemented The steps of the method described in item.

In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program, when executed by a processor, implements the steps of any one of the methods in the first aspect.

This application provides a data transmission method. This method is applied to the data channel system. The data channel system includes a gateway control module and a data channel cluster. Among them, the gateway control module is connected to the database, and the data channel cluster is connected to the application system. The gateway control module receives the first data in the first format sent by the database, and sends the first data to the data channel cluster. Wherein, the first format is a format in which the first data is stored in the database. The data channel cluster converts the first format of the first data into the second format, obtains the first data after the format conversion, and sends the first data after the format conversion to the application system. Wherein, the second format is a format in which the first data is stored in the application system. The data channel cluster receives the second data in the third format sent by the application system, converts the third format of the second data into the fourth format, obtains the second data after the format conversion, and sends the second data after the format conversion to Gateway control module. Among them, the third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database. The gateway control module sends the converted second data to the database. In this way, there is no need to spend manpower and material resources on the secondary development of the application system. By setting a data channel system between the application system and the database, data interaction between the application system and the database is realized, and the database is the hub to realize the interaction between different application systems. .

Description of the drawings

FIG. 1 is an architecture diagram of an application environment provided by an embodiment of the application;

FIG. 2 is a flowchart of a data transmission method provided by an embodiment of this application;

FIG. 3 is a flowchart of a method for creating a data channel cluster provided by an embodiment of the application;

FIG. 4 is a flowchart of a metadata management method provided by an embodiment of the application;

FIG. 5A is a flowchart of an example of a data transmission method provided by an embodiment of this application;

5B is a flowchart of an example of a data transmission method provided by an embodiment of this application;

FIG. 6 is a schematic structural diagram of a data channel system provided by an embodiment of this application;

FIG. 7 is a schematic structural diagram of a computer device provided by an embodiment of this application.

Detailed ways

In order to make the purpose, technical solutions, and advantages of this application clearer and clearer, the following further describes the application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

The embodiment of the application provides a data transmission method, which can be applied to a data channel system in an application scenario of data interaction between an application system and a database. FIG. 1 is an architecture of an application environment provided by an embodiment of the application. picture. As shown in FIG. 1, the application environment includes a database 110, a data channel system 120 and an application system 130. The database 110 and the application system 130 are connected through a data channel system 120. Among them, the database 110 is used to store video image data resources (such as face information, vehicle information, alarm information, collection equipment information, etc.) of the application system based on a standard format (such as GAT1400 format); the data channel system 120 is used to check the database The video image data transmitted between 110 and the application system 130 undergoes format conversion processing; the application system 130 is used to obtain the video image data resources uploaded by other application systems 130 from the database 110 through the data channel system 120, or through the data channel system 120 uploads the locally stored video image data resource to the database 110 so that other application systems 130 can obtain the video image data resource from the database 110.

In the following, a data transmission method provided by an embodiment of the present application will be described in detail in conjunction with specific implementations. The method is applied to a data channel system, and the data channel system includes a gateway control module and a data channel cluster. Among them, the gateway control module is connected to the database, and the data channel cluster is connected to the application system. As shown in Figure 2, the specific steps are as follows:

Step 201: The gateway control module receives the first data in the first format sent by the database, and sends the first data to the data channel cluster.

Wherein, the first format is a format in which the first data is stored in the database.

In the implementation, because the types of data collected by each application system may be different, the database stores various types of data uploaded by each application system (such as snapshot data, alarm data, equipment data, etc.), and different types of data correspond to For different formats, the data in the first format is recorded as the first data. When a certain application system needs to obtain a certain type of data (that is, the first data) uploaded by other application systems from the database, the application system may send a data request to the database. Wherein, the data request carries the address of the first data. After receiving the data request, the database can send the first data in the first format to the data channel system. Wherein, the first format is a format in which the first data is stored in the database. Correspondingly, the gateway control module in the data channel system can receive the first data in the first format sent by the database, and send the first data to the data channel cluster.

As an optional implementation manner, the gateway control module includes an application program interface unit, the application program interface unit includes an application program interface corresponding to the first data, and the data channel cluster includes a first data channel corresponding to the first data. The gateway control module receives the first data in the first format sent by the database, and sends the first data to the data channel cluster. The processing process is that the application program interface unit receives the first data in the first format sent by the database through the first application program interface. , And send the first data to the first data channel in the data channel cluster.

In implementation, the gateway control module includes an application program interface unit. In order to distinguish different types of data, the application program interface unit includes application program interfaces for receiving different types of data (that is, one application program interface in the application program interface unit corresponds to one type of data). At the same time, because different types of data are processed differently. Therefore, the data channel cluster includes data channels for processing different types of data (that is, one data channel in the data channel cluster corresponds to one type of data). Therefore, a type of data corresponds to an application program interface in the application program interface unit, and corresponds to a data channel in the data channel cluster (for example, snapshot data corresponds to the snapshot data channel, alarm data corresponds to the alarm data channel, and device data corresponds to the device data Channels, etc.). Correspondingly, for the first data, the application program interface unit includes the first application program interface for the first data, and the data channel cluster includes the first data channel corresponding to the first data. In this way, the application program interface unit in the gateway control module can receive the first data in the first format sent by the database through the first application program interface. Then, the application program interface unit may send the first data to the first data channel in the data channel cluster.

Step 202: The data channel cluster converts the first format of the first data into the second format, obtains the first data after the format conversion, and sends the first data after the format conversion to the application system.

Wherein, the second format is a format in which the first data is stored in the application system.

In the implementation, since the application system mainly focuses on the realization of the video image application function in the early stage of development, it does not support the standard specification, that is, the format of the first data stored in the application system and the database is not necessarily the same. Therefore, in order to ensure that the application system can store and process the first data, after receiving the first data, the data channel cluster can further convert the first format of the first data into the second format to obtain the first data after the format conversion. Wherein, the second format is a format in which the first data is stored in the application system. Then, the data channel cluster can send the first data after the format conversion to the application system. Correspondingly, the application system can receive the first data in the second format for storage and processing. In this way, there is no need to spend manpower and material resources on the secondary development of the application system, and data interaction between the application system and the database is realized by setting a data channel system between the application system and the database.

As an optional implementation manner, the data channel cluster includes a first data channel corresponding to the first data, and the first data channel includes a first data mapping table and a first data driver corresponding to the first data. The data channel cluster converts the first format of the first data into the second format, obtains the first data after the format conversion, and sends the first data after the format conversion to the application system. The processing process is as follows.

Step 1: The first data channel converts the first format of the first data into the second format according to the first data mapping table to obtain the first data after the format conversion.

In the implementation, due to the different format conversion processing of different types of data, the storage methods of different types of data in the application system are also different, and the data drivers corresponding to different storage methods are also different (such as Representational State Transfer, REST ) The interface corresponds to the HyperText Transfer Protocol (HTTP) client, the database corresponds to the database connection pool, and the message middleware corresponds to the consumer listener). Therefore, the data channel cluster contains data channels for processing different types of data. Among them, the data channel includes a data mapping table for format conversion of different types of data and a data driver for sending different types of data. Correspondingly, for the first data, the data channel cluster includes a first data channel corresponding to the first data, and the first data channel includes a first data mapping table and a first data driver corresponding to the first data. In this way, after receiving the first data, the first data channel can convert the first format of the first data into the second format according to the first data mapping table to obtain the first data after the format conversion.

Step 2: The first data channel sends the converted first data to the application system through the first data driver.

In implementation, after the first data channel obtains the first data after the format conversion, the first data after the format conversion may be sent to the application system through the first data driver. Correspondingly, the application system can receive the first data in the second format for storage and processing.

Step 203: The data channel cluster receives the second data in the third format sent by the application system, converts the third format of the second data into the fourth format, obtains the second data after the format conversion, and converts the second data after the format conversion. The data is sent to the gateway control module.

Among them, the third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database.

In implementation, when an application system needs to upload a certain type of locally stored data (ie, second data) to the database, the application system can send the second format of the third format to the data channel cluster in the data channel system. data. Among them, the third format is a format for storing the second data in the application system. Correspondingly, the data channel cluster in the data channel system can receive the second data in the third format sent by the application system. Because the format of the second data stored in the application system and the database is not necessarily the same. Therefore, in order to ensure that the database can store the second data, the data channel cluster may further convert the third format of the second data into the fourth format to obtain the second data after the format conversion. Among them, the fourth format is a format for storing the second data in the database. After that, the data channel cluster can send the second data after the format conversion to the gateway control module.

As an optional implementation manner, the data channel cluster includes a second data channel corresponding to the second data, and the second data channel includes a second data mapping table and a second data driver corresponding to the second data. The data channel cluster receives the second data in the third format sent by the application system, converts the third format of the second data into the fourth format, obtains the second data after the format conversion, and sends the second data after the format conversion to The processing procedure of the gateway control module is as follows.

Step 1: The second data channel receives the second data in the third format sent by the application system through the second data driver.

In the implementation, because the format conversion processing of different types of data is different, the storage methods of different types of data in the application system are different, and the data drivers corresponding to different storage methods are also different (for example, Representational State Transfer, REST) The interface corresponds to the HyperText Transfer Protocol (HTTP) client, the database corresponds to the database connection pool, and the message middleware corresponds to the consumer listener). Therefore, the data channel cluster contains data channels for processing different types of data. Among them, the data channel includes a data mapping table for format conversion of different types of data and a data driver for receiving different types of data. Correspondingly, for the second data, the data channel cluster includes a second data channel corresponding to the second data, and the second data channel includes a second data mapping table and a second data driver corresponding to the second data. In this way, the application system can send the second data in the third format to the second data driver of the second data channel in the data channel cluster. Correspondingly, the second data channel can receive the second data in the third format sent by the application system through the second data driver.

Step 2: The second data channel converts the third format of the second data into the fourth format according to the second data mapping table to obtain the second data after the format conversion, and sends the second data after the format conversion to the gateway control module .

In implementation, after the second data channel receives the second data in the third format sent by the application system, it can convert the third format of the second data into the fourth format according to the second data mapping table to obtain the converted format The second data. Then, the second data channel may send the second data after the format conversion to the gateway control module.

Step 204: The gateway control module sends the converted second data to the database.

In implementation, after the gateway control module receives the format-converted second data, it can send the format-converted second data to the database. Correspondingly, the database can receive the second data in the fourth format to store the second data in the fourth format. In this way, there is no need to spend manpower and material resources on the secondary development of the application system, and data interaction between the application system and the database is realized by setting a data channel system between the application system and the database.

As an optional implementation manner, the gateway control module includes a gateway unit. The processing process of the gateway control module sending the second data after the format conversion to the database is that the gateway control module sends the second data after the format conversion to the database through the gateway unit.

In implementation, the gateway control module also includes a gateway unit for sending data to the database. After the gateway control module receives the format-converted second data, it can send the format-converted second data to the database through the gateway unit.

It should be noted that step 101 and step 102 are processes for the application system to obtain the first data from the database through the data channel system, and steps 103 and 104 are processes for the application system to upload the second data to the database through the data channel. The processing sequence of the above two processes does not distinguish the sequence.

As an optional implementation manner, the data channel system further includes a channel management module, and the channel management module is used to create a data channel cluster. As shown in Figure 3, the specific processing process is as follows:

Step 301: The channel management module obtains the data driver and the data mapping table corresponding to each data.

In implementation, because different types of data have different format conversion processing, and different types of data are stored in different ways in the application system. Therefore, data drivers and data mapping tables corresponding to different types of data are also different (that is, one type of data corresponds to one data driver and one data mapping table). Data drivers and data mapping tables corresponding to various types of data can be pre-stored in the data channel system. Among them, the generation process of the data driver and the data mapping table corresponding to each type of data will be described in detail later. When the channel management module needs to create data channel clusters corresponding to various types of data, the channel management module can obtain the data corresponding to each type from the data channel system pre-stored with data drivers and data mapping tables corresponding to each type of data. Data drive and data mapping table. Optionally, the channel management module can create a data channel cluster when the data channel system is initialized, or it can create a data channel cluster according to a user's instruction, which is not limited in this application.

In step 302, the channel management module creates a data channel corresponding to each data according to the data driver and the data mapping table corresponding to each data to form a data channel cluster.

In implementation, since one type of data corresponds to a program interface in the application program interface unit, and a data channel in the data channel cluster, the channel management module obtains the data driver and data map corresponding to each type of data. After the table, for each type of data, the channel management module can create a data channel corresponding to the type of data according to the data driver and data mapping table corresponding to the type of data, so as to obtain the data channel corresponding to each type of data. Data channel cluster. Optionally, the channel management module can also open a data channel in the data channel cluster, pause operation, filter setting, traffic monitoring setting, destroy operation, re-creation operation, and data channel cluster copy operation according to the user's instructions. . In addition, the channel management module can also provide a fuse protection strategy for the data channel when any one of the application system and the database is down abnormally.

As an optional implementation manner, the data channel system includes multiple data channel clusters, and the multiple data channel clusters include a main data channel cluster and a backup data channel cluster. The channel management module can

When the channel management module detects the failure of the first data channel cluster, the second data channel cluster is switched to the primary data channel cluster, the first data channel cluster is the primary data channel cluster, and the second data channel cluster is the backup data channel cluster.

As an optional implementation manner, the data channel system further includes a metadata management module. The metadata management module can generate the drive and data mapping table corresponding to each data according to the metadata of each data. As shown in Figure 4, the specific process is as follows.

Step 401: The metadata management module obtains metadata of each data.

In implementation, the data channel system may pre-store metadata of each view data resource (that is, each data) in the application system. Among them, metadata is a method used to describe the acquisition of various view data resources (such as face information, vehicle information, alarm information, collection equipment information, etc.) in the application system (such as REST interface, message middleware, database, big data, etc.) Warehouse, etc.) data rules; metadata includes two parts: data sources (resources) and field mappings (mappings). In order to facilitate processing, technicians can describe and configure the metadata of each data in XML (an extensible markup language) or JSON (a programming language) in advance. For example, the metadata of the captured data resource includes the mapping relationship between the captured data source and the captured field. When the metadata management module needs to generate the drive and data mapping table corresponding to each data, the metadata management module can obtain the metadata of each data.

In step 402, the metadata management module parses the metadata of each data to obtain data driver metadata and data mapping table metadata corresponding to each data.

In implementation, after the metadata management module obtains the metadata of each data, for each data metadata, the metadata management module can analyze and process the metadata of the data to obtain the data driver metadata and data corresponding to the data. Data mapping table metadata. Among them, the data driver metadata corresponds to the data source in the metadata, and the data mapping table metadata corresponds to the field mapping relationship in the metadata.

Step 403: The metadata management module translates the data driver metadata and data mapping table metadata corresponding to each data to obtain the data driver and data mapping table corresponding to each data, and stores the data driver and data mapping table corresponding to each data. .

In the implementation, after the metadata management module obtains the data driver metadata and data mapping table metadata corresponding to each data, for each data, the metadata management module can the data driver metadata and data mapping table metadata corresponding to the data Perform translation processing to obtain the data driver and data mapping table corresponding to the data. Then, the metadata management module can store the data driver and data mapping table corresponding to each data locally. Optionally, the metadata management module can store the data driver and data mapping table corresponding to each data in ETCD (an open source, high-availability, high-concurrency distributed key-value storage system), so as to ensure the creation of data channels The data driver and data mapping table will not be lost, thereby improving the stability of the channel system. In addition, the data channel corresponding to each type of data can be deployed in multiple instances. Through the management of the data driver and data mapping table of the data through ETCD, the data idempotence between multiple instances can be ensured.

FIG. 5A is a flowchart of an example of a data transmission method provided by an embodiment of the application. In this example, we take the snapshot data as an example. The snapshot data corresponds to the snapshot application program interface in the application program interface unit, and the data channel cluster corresponds to the snapshot data channel. The snapshot data channel includes the snapshot data driver and the snapshot data mapping table. . As shown in Fig. 5A, the processing procedure includes the following steps.

Step 511: The application program interface unit in the gateway control module receives the snapshot data in the first format sent by the database through the snapshot application program interface. Among them, the first format is the format for storing the snapshot data in the database.

Step 512: The application program interface unit in the gateway control module sends the snapshot data in the first format to the snapshot data channel in the data channel cluster.

In step 513, the captured data channel converts the first format of the captured data into the second format through the captured data mapping table to obtain the captured captured data after the format conversion. Among them, the second format is a format for storing snapshot data in the application system.

In step 514, the snapshot data channel sends the format-converted snapshot data to the application system through the snapshot data driver.

The processing procedure from step 511 to step 514 and the processing procedure type from step 201 to step 202 are not repeated here.

FIG. 5B is a flowchart of an example of a data transmission method provided by an embodiment of the application. In this example, the snapshot data is taken as an example. The snapshot data corresponds to the snapshot data channel in the data channel cluster. The snapshot data channel includes the snapshot data driver and the snapshot data mapping table. As shown in FIG. 5B, the processing procedure includes the following steps.

Step 521: The snapshot data channel in the data channel cluster receives the snapshot data in the second format sent by the application system through the snapshot data driver. Among them, the second format is a format for storing snapshot data in the application system.

Step 522: The captured data channel converts the second format of the captured data into the first format through the captured data mapping table to obtain the captured captured data after the format conversion. Among them, the first format is the format for storing the snapshot data in the database.

Step 523: The snapshot data channel sends the format-converted snapshot data to the gateway unit in the gateway control module.

In step 524, the gateway control module sends the converted snapshot data to the database through the gateway unit.

The processing procedure from step 521 to step 524 and the processing procedure type from step 203 to step 204 are not repeated here.

The embodiment of the present application provides a data transmission method. This method is applied to the data channel system. The data channel system includes a gateway control module and a data channel cluster. Among them, the gateway control module is connected to the database, and the data channel cluster is connected to the application system. The gateway control module receives the first data in the first format sent by the database, and sends the first data to the data channel cluster. Wherein, the first format is a format in which the first data is stored in the database. The data channel cluster converts the first format of the first data into the second format, obtains the first data after the format conversion, and sends the first data after the format conversion to the application system. Wherein, the second format is a format in which the first data is stored in the application system. The data channel cluster receives the second data in the third format sent by the application system, converts the third format of the second data into the fourth format, obtains the second data after the format conversion, and sends the second data after the format conversion to Gateway control module. Among them, the third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database. The gateway control module sends the converted second data to the database. In this way, there is no need to spend manpower and material resources on the secondary development of the application system. By setting a data channel system between the application system and the database, data interaction between the application system and the database is realized, and the database is the hub to realize the interaction between different application systems. .

An embodiment of the present application also provides a data channel system. As shown in FIG. 6, the data channel system includes a gateway control module 610 and a data channel cluster 620. The gateway control module 610 is connected to a database, and the data channel cluster 620 is connected to the application system;

The gateway control module 610 is configured to receive first data in a first format sent by the database, and send the first data to the data channel cluster 620; the first format is stored in the database The format of the first data;

The data channel cluster 620 is configured to convert the first format of the first data into a second format, obtain the first data after the format conversion, and send the first data after the format conversion to the The application system; the second format is the format in which the first data is stored in the application system;

The data channel cluster 620 is configured to receive second data in a third format sent by the application system, convert the third format of the second data into a fourth format, and obtain the second data after the format conversion , And send the second data after the format conversion to the gateway control module 610; the third format is the format in which the second data is stored in the application system, and the fourth format is the database The format in which the second data is stored in;

The gateway control module 610 is configured to send the second data after the format conversion to the database.

As an optional implementation manner, the gateway control module 610 includes an application program interface unit, the application program interface unit includes an application program interface corresponding to the first data, and the data channel cluster 620 includes the first The first data channel corresponding to the data;

The application program interface unit is configured to receive the first data in the first format sent by the database through the first application program interface, and send the first data to the data channel cluster 620. The first data channel.

As an optional implementation manner, the data channel cluster 620 includes a first data channel corresponding to the first data, and the first data channel includes a first data mapping table and a first data mapping table corresponding to the first data. Data drive

As an optional implementation manner, the data channel cluster 620 includes a second data channel corresponding to the second data, and the second data channel includes a second data mapping table and a second data mapping table corresponding to the second data. Data drive

The second data channel is configured to convert the third format of the second data into a fourth format according to the second data mapping table to obtain the second data after the format conversion, and convert the format The latter second data is sent to the gateway control module 610.

As an optional implementation manner, the gateway control module 610 includes a gateway unit;

The gateway control module 610 is configured to send the second data after the format conversion to the database through the gateway unit.

As an optional implementation manner, the data channel system further includes a channel management module 630;

The channel management module 630 is used to obtain the data driver and data mapping table corresponding to each data;

The channel management module 630 is configured to create a data channel corresponding to each data according to a data driver and a data mapping table corresponding to each data to form the data channel cluster 620.

The channel management module 630 is configured to switch the second data channel cluster to the primary data channel cluster when the failure of the first data channel cluster is detected, the first data channel cluster is the primary data channel cluster, and the second data channel cluster is the primary data channel cluster. The data channel cluster is a backup data channel cluster.

As an optional implementation manner, the data channel system further includes a metadata management module 640;

The metadata management module 640 is used to obtain metadata of each data;

The metadata management module 640 is configured to analyze and process the metadata of each data to obtain data driver metadata and data mapping table metadata corresponding to each data;

The metadata management module 640 is configured to perform translation processing on the data driver metadata and data mapping table metadata corresponding to each data to obtain the data driver and data mapping table corresponding to each data, and to store the data The data driver and data mapping table corresponding to the data.

The embodiment of the present application provides a data channel system. The data channel system includes a gateway control module and a data channel cluster. Among them, the gateway control module is connected to the database, and the data channel cluster is connected to the application system. The gateway control module receives the first data in the first format sent by the database, and sends the first data to the data channel cluster. Wherein, the first format is a format in which the first data is stored in the database. The data channel cluster converts the first format of the first data into the second format, obtains the first data after the format conversion, and sends the first data after the format conversion to the application system. Wherein, the second format is a format in which the first data is stored in the application system. The data channel cluster receives the second data in the third format sent by the application system, converts the third format of the second data into the fourth format, obtains the second data after the format conversion, and sends the second data after the format conversion to Gateway control module. Among them, the third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database. The gateway control module sends the converted second data to the database. In this way, there is no need to spend manpower and material resources on the secondary development of the application system. By setting a data channel system between the application system and the database, data interaction between the application system and the database is realized, and the database is the hub to realize the interaction between different application systems. .

In one embodiment, a computer device is provided, as shown in FIG. 6, including a memory and a processor, the memory stores a computer program that can run on the processor, and the processor executes the computer program When realizing the above-mentioned data transmission method.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program is executed by a processor to realize the above-mentioned data transmission method.

A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage. In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and/or volatile memory. Non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not a limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous chain Channel (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be It is considered as the range described in this specification.

The above-mentioned embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the invention patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can be made, and these all fall within the scope of protection of this application. Therefore, the scope of protection of the patent of this application shall be subject to the appended claims.

Claims

A data transmission method, characterized in that the method is applied to a data channel system, the data channel system includes a gateway control module and a data channel cluster, the gateway control module is connected to a database, and the data channel cluster is connected to the application System connection, the method includes:

The gateway control module receives the first data in the first format sent by the database, and sends the first data to the data channel cluster; the first format is that the first data is stored in the database Format

The data channel cluster converts the first format of the first data into a second format, obtains the first data after the format conversion, and sends the first data after the format conversion to the application system; The second format is a format in which the first data is stored in the application system;

The data channel cluster receives the second data in the third format sent by the application system, converts the third format of the second data into a fourth format, obtains the second data after the format conversion, and combines The second data after the format conversion is sent to the gateway control module; the third format is the format for storing the second data in the application system, and the fourth format is for storing the first data in the database. 2. The format of the data;

The gateway control module sends the second data after the format conversion to the database.
The method according to claim 1, wherein the gateway control module includes an application program interface unit, the application program interface unit includes an application program interface corresponding to the first data, and the data channel cluster includes the The first data channel corresponding to the first data;

The gateway control module receiving the first data in the first format sent by the database and sending the first data to the data channel cluster includes:

The application program interface unit receives the first data in the first format sent by the database through the first application program interface, and sends the first data to the first data channel in the data channel cluster .
The method according to claim 1 or 2, wherein the data channel cluster includes a first data channel corresponding to the first data, and the first data channel includes first data corresponding to the first data The mapping table and the first data driver;

The data channel cluster converts the first format of the first data into a second format, obtains the first data after the format conversion, and sends the first data after the format conversion to the application system, include:

Converting, by the first data channel, the first format of the first data into a second format according to the first data mapping table, to obtain the first data after the format conversion;

The first data channel sends the first data after the format conversion to the application system through the first data driver.
The method according to claim 1, wherein the data channel cluster includes a second data channel corresponding to the second data, and the second data channel includes a second data mapping table corresponding to the second data And the second data drive;

The data channel cluster receives the second data in the third format sent by the application system, converts the third format of the second data into a fourth format, obtains the second data after the format conversion, and combines The second data after the format conversion is sent to the gateway control module, including:

The second data channel receives the second data in the third format sent by the application system through the second data driver;

The second data channel converts the third format of the second data into a fourth format according to the second data mapping table to obtain the second data after the format conversion, and the second data after the format conversion is obtained. The second data is sent to the gateway control module.
The method according to claim 1 or 4, wherein the gateway control module comprises a gateway unit;

The gateway control module sending the second data after the format conversion to the database includes:

The gateway control module sends the second data after the format conversion to the database through the gateway unit.
The method according to claim 1, wherein the data channel system further comprises a channel management module, and the method further comprises:

The channel management module obtains the data driver and the data mapping table corresponding to each data;

The channel management module creates a data channel corresponding to each data according to a data driver and a data mapping table corresponding to each data to form the data channel cluster.
The method according to claim 6, wherein the data channel system includes multiple data channel clusters, the multiple data channel clusters including a primary data channel cluster and a backup data channel cluster, and the method further comprises:

When the channel management module detects the failure of the first data channel cluster, the second data channel cluster is switched to the main data channel cluster, the first data channel cluster is the main data channel cluster, and the second data channel cluster is Backup data channel cluster.
The method according to claim 1, wherein the data channel system further comprises a metadata management module, and the method further comprises:

The metadata management module obtains metadata of each data;

The metadata management module analyzes and processes the metadata of each data to obtain data driver metadata and data mapping table metadata corresponding to each data;

The metadata management module performs translation processing on the data driver metadata and data mapping table metadata corresponding to each data to obtain the data driver and data mapping table corresponding to each data, and stores the data corresponding to each data Drive and data mapping table.
A data channel system, characterized in that the data channel system includes a gateway control module and a data channel cluster, the gateway control module is connected to a database, and the data channel cluster is connected to an application system;

The gateway control module is configured to receive first data in a first format sent by the database, and send the first data to the data channel cluster; the first format is that the database stores the The format of the first data;

The data channel cluster is configured to convert the first format of the first data into a second format, obtain the first data after the format conversion, and send the first data after the format conversion to the Application system; the second format is a format in which the first data is stored in the application system;

The data channel cluster is configured to receive second data in a third format sent by the application system, convert the third format of the second data into a fourth format, and obtain the second data after the format conversion, And send the second data after the format conversion to the gateway control module; the third format is the format in which the second data is stored in the application system, and the fourth format is the format stored in the database The format of the second data;

The gateway control module is configured to send the second data after the format conversion to the database.
A computer device, comprising a memory and a processor, and a computer program that can be run on the processor is stored on the memory, wherein the processor implements any one of claims 1 to 8 when the computer program is executed. The steps of the method described in item.
A computer-readable storage medium with a computer program stored thereon, wherein the computer program implements the steps of the method according to any one of claims 1 to 8 when the computer program is executed by a processor.