CN111405035A - Data transmission method and data channel system - Google Patents
Data transmission method and data channel system Download PDFInfo
- Publication number
- CN111405035A CN111405035A CN202010174731.3A CN202010174731A CN111405035A CN 111405035 A CN111405035 A CN 111405035A CN 202010174731 A CN202010174731 A CN 202010174731A CN 111405035 A CN111405035 A CN 111405035A
- Authority
- CN
- China
- Prior art keywords
- data
- format
- data channel
- channel
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/565—Conversion or adaptation of application format or content
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L12/00—Data switching networks
- H04L12/66—Arrangements for connecting between networks having differing types of switching systems, e.g. gateways
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer And Data Communications (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to a data transmission method and a data channel system. The method is applied to a data channel system and comprises the following steps: the gateway control module receives first data in a first format sent by a database and sends the first data to the data channel cluster; the data channel cluster converts a first format of the first data into a second format to obtain first data after format conversion, and sends the first data after format conversion to an application system; the data channel cluster receives second data in a third format sent by the application system, converts the third format of the second data into a fourth format to obtain second data after format conversion, and sends the second data after format conversion to the gateway control module; and the gateway control module sends the second data after format conversion to a database. By the method and the device, data interaction between the application system and the database can be realized.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data transmission method and a data channel system.
Background
At present, video image information application systems (hereinafter referred to as application systems) are widely applied in the security industry, and the variety of the video image information application systems is more and more. However, the application systems usually do not have a data transceiving function, so that data cannot be transmitted between different application systems, and further coordination work in video image application cannot be performed between security units.
In order to solve the problems, the ministry of public security issues a standard specification of GAT1400 public security video image information application system. In the standard specification, a video image information database (hereinafter referred to as a database) is used as a hub of data interaction, and each application system realizes data interaction through the database.
However, the application system mainly focuses on the implementation of the video image application function in the early development stage, and is not supported by the standard specification and the database. Therefore, when the application system needs to perform data interaction with the database, the application system needs to be developed secondarily by consuming manpower and material resources.
Disclosure of Invention
In view of the above, it is necessary to provide a data transmission method and a data channel system for solving the above technical problems.
In a first aspect, a data transmission method is provided, where the method is applied to a data channel system, where the data channel system includes a gateway control module and a data channel cluster, the gateway control module is connected to a database, and the data channel cluster is connected to an application system, and the method includes:
the gateway control module receives first data in a first format sent by the database and sends the first data to the data channel cluster; the first format is a format in which the first data is stored in the database;
the data channel cluster converts the first format of the first data into a second format to obtain first data after format conversion, and sends the first data after format conversion to the application system; the second format is a format for storing the first data in the application system;
the data channel cluster receives second data in a third format sent by the application system, converts the third format of the second data into a fourth format to obtain second data with converted formats, and sends the second data with converted formats to the gateway control module; the third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database;
and the gateway control module sends the second data after format conversion to the database.
As an optional implementation manner, the gateway control module includes an application program interface unit, where the application program interface unit includes an application program interface corresponding to the first data, and the data channel cluster includes a first data channel corresponding to the first data;
the gateway control module receives first data in a first format sent by the database, and sends the first data to the data channel cluster, and the method comprises the following steps:
and the application program interface unit receives first data in a first format sent by the database through the first application program interface and sends the first data to the first data channel in the data channel cluster.
As an optional implementation manner, the data channel cluster includes a first data channel corresponding to the first data, where the first data channel includes a first data mapping table and a first data driver corresponding to the first data;
the data channel cluster converts the first format of the first data into a second format to obtain first data after format conversion, and sends the first data after format conversion to the application system, and the method comprises the following steps:
the first data channel converts the first format of the first data into a second format according to the first data mapping table to obtain first data after format conversion;
and the first data channel sends the format-converted first data to the application system through the first data driver.
As an optional implementation manner, the data channel cluster includes a second data channel corresponding to the second data, where the second data channel includes a second data mapping table and a second data driver corresponding to the second data;
the data channel cluster receives second data in a third format sent by the application system, converts the third format of the second data into a fourth format to obtain second data after format conversion, and sends the second data after format conversion to the gateway control module, and the method includes:
the second data channel receives second data in a third format sent by the application system through the second data driver;
and the second data channel converts the third format of the second data into a fourth format according to the second data mapping table to obtain second data with converted formats, and sends the second data with converted formats to the gateway control module.
As an optional implementation, the gateway control module includes a gateway unit;
the gateway control module sends the second data after format conversion to the database, and the method comprises the following steps:
and the gateway control module sends the second data after format conversion to the database through the gateway unit.
As an optional implementation, the data channel system further includes a channel management module, and the method further includes:
the channel management module acquires a data driver and a data mapping table corresponding to each data;
and the channel management module creates a data channel corresponding to each data according to the data driver corresponding to each data and the data mapping table to form the data channel cluster.
As an optional implementation, the data tunnel system includes a plurality of data tunnel clusters, where the plurality of data tunnel clusters includes a primary data tunnel cluster and a backup data tunnel cluster, and the method further includes:
when the channel management module detects that a first data channel cluster fails, a second data channel cluster is switched to a main data channel cluster, the first data channel cluster is a main data channel cluster, and the second data channel cluster is a backup data channel cluster.
As an optional implementation, the data channel system further includes a metadata management module, and the method further includes:
the metadata management module acquires metadata of each data;
the metadata management module analyzes the metadata of each data to obtain the metadata of a data driver and the metadata of a data mapping table corresponding to each data;
and the metadata management module translates the metadata of the data driver and the metadata of the data mapping table corresponding to each data to obtain the data driver and the data mapping table corresponding to each data, and stores the data driver and the data mapping table corresponding to each data.
In a second aspect, a data channel system is provided, where the data channel system includes a gateway control module and a data channel cluster, the gateway control module is connected to a database, and the data channel cluster is connected to an application system;
the gateway control module is used for receiving first data in a first format sent by the database and sending the first data to the data channel cluster; the first format is a format in which the first data is stored in the database;
the data channel cluster is used for converting the first format of the first data into a second format to obtain first data after format conversion, and sending the first data after format conversion to the application system; the second format is a format for storing the first data in the application system;
the data channel cluster is used for receiving second data in a third format sent by the application system, converting the third format of the second data into a fourth format to obtain second data after format conversion, and sending the second data after format conversion to the gateway control module; the third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database;
and the gateway control module is used for sending the second data after the format conversion to the database.
As an optional implementation manner, the gateway control module includes an application program interface unit, where the application program interface unit includes an application program interface corresponding to the first data, and the data channel cluster includes a first data channel corresponding to the first data;
the application program interface unit is configured to receive, through the first application program interface, first data in a first format sent by the database, and send the first data to the first data channel in the data channel cluster.
As an optional implementation manner, the data channel cluster includes a first data channel corresponding to the first data, where the first data channel includes a first data mapping table and a first data driver corresponding to the first data;
the first data channel is used for converting the first format of the first data into a second format according to the first data mapping table to obtain format-converted first data;
the first data channel is used for sending the first data after format conversion to the application system through the first data driver.
As an optional implementation manner, the data channel cluster includes a second data channel corresponding to the second data, where the second data channel includes a second data mapping table and a second data driver corresponding to the second data;
the second data channel is used for receiving second data in a third format sent by the application system through the second data driver;
the second data channel is configured to convert the third format of the second data into a fourth format according to the second data mapping table, obtain second data with a converted format, and send the second data with the converted format to the gateway control module.
As an optional implementation, the gateway control module includes a gateway unit;
and the gateway control module is used for sending the second data after the format conversion to the database through the gateway unit.
As an optional implementation manner, the data channel system further includes a channel management module;
the channel management module is used for acquiring a data driver and a data mapping table corresponding to each data;
and the channel management module is used for creating a data channel corresponding to each data according to the data driver corresponding to each data and the data mapping table to form the data channel cluster.
As an optional implementation manner, the data channel system includes a plurality of data channel clusters, where the plurality of data channel clusters include a main data channel cluster and a backup data channel cluster;
the channel management module is configured to switch a second data channel cluster to a main data channel cluster when detecting that a first data channel cluster fails, where the first data channel cluster is the main data channel cluster, and the second data channel cluster is a backup data channel cluster.
As an optional implementation, the data channel system further includes a metadata management module;
the metadata management module is used for acquiring metadata of each data;
the metadata management module is used for analyzing and processing the metadata of each data to obtain the metadata of a data driver and the metadata of a data mapping table corresponding to each data;
and the metadata management module is used for translating the metadata of the data driver and the metadata of the data mapping table corresponding to each data to obtain the data driver and the data mapping table corresponding to each data, and storing the data driver and the data mapping table corresponding to each data.
In a third aspect, a computer device is provided, comprising a memory and a processor, the memory having stored thereon a computer program operable on the processor, the processor implementing the steps of the method of any one of the first aspect when executing the computer program.
In a fourth aspect, a computer-readable storage medium is provided, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the method of any one of the first aspects.
The application provides a data transmission method. The method is applied to a data channel system. The data channel system comprises a gateway control module and a data channel cluster. The gateway control module is connected with the database, and the data channel cluster is connected with the application system. The gateway control module receives first data in a first format sent by the database and sends the first data to the data channel cluster. The first format is a format for storing first data in the database. The data channel cluster converts the first format of the first data into the second format to obtain the first data after format conversion, and sends the first data after format conversion to the application system. The second format is a format for storing the first data in the application system. And the data channel cluster receives second data in a third format sent by the application system, converts the third format of the second data into a fourth format to obtain second data after format conversion, and sends the second data after format conversion to the gateway control module. The third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database. And the gateway control module sends the second data after format conversion to a database. Therefore, secondary development of the application system is not required to be performed by consuming manpower and material resources, and the data channel system is arranged between the application system and the database, so that data interaction between the application system and the database is realized, and interaction between different application systems is realized by taking the database as a hub.
Drawings
Fig. 1 is an architecture diagram of an application environment according to an embodiment of the present application;
fig. 2 is a flowchart of a data transmission method according to an embodiment of the present application;
fig. 3 is a flowchart of a method for creating a data channel cluster according to an embodiment of the present application;
fig. 4 is a flowchart of a method for managing metadata according to an embodiment of the present application;
fig. 5A is a flowchart of an example of a data transmission method according to an embodiment of the present application;
fig. 5B is a flowchart of an example of a data transmission method according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of a data channel system according to an embodiment of the present application;
fig. 7 is a schematic structural diagram of a computer device according to an embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The embodiment of the present application provides a data transmission method, which may be applied to a data channel system in an application scenario of data interaction between an application system and a database, and fig. 1 is an architecture diagram of an application environment provided in the embodiment of the present application. As shown in FIG. 1, the application environment includes a database 110, a data channel system 120, and an application system 130. The database 110 is connected with the application system 130 through a data channel system 120. The database 110 is configured to store video image data resources (such as face information, vehicle information, warning information, collecting device information, and the like) of the application system based on a standard format (such as the GAT1400 format); a data channel system 120 for performing format conversion processing on the video image data transmitted between the database 110 and the application system 130; the application system 130 is configured to obtain the video image data resource uploaded by the other application system 130 from the database 110 through the data channel system 120, or upload the locally stored video image data resource to the database 110 through the data channel system 120, so that the other application system 130 can obtain the video image data resource from the database 110.
The following describes, in detail, a data transmission method provided in an embodiment of the present application with reference to a specific embodiment, where the method is applied to a data channel system, and the data channel system includes a gateway control module and a data channel cluster. The gateway control module is connected with the database, and the data channel cluster is connected with the application system. As shown in fig. 2, the specific steps are as follows:
step 201, the gateway control module receives first data in the first format sent by the database, and sends the first data to the data channel cluster.
The first format is a format for storing first data in the database.
In implementation, because types of data collected by each application system may be different, various types of data (such as snapshot data, alarm data, device data, and the like) uploaded by each application system are stored in the database, and the different types of data correspond to different formats, and the data in the first format is recorded as the first data. When an application system needs to obtain a certain type of data (i.e., first data) uploaded by other application systems from a database, the application system may send a data request to the database. The data request carries an address of the first data. The database may send the first data in the first format to the data channel system after receiving the data request. The first format is a format for storing first data in the database. Correspondingly, the gateway control module in the data channel system may receive the first data in the first format sent by the database, and send the first data to the data channel cluster.
As an optional implementation manner, the gateway control module includes an application program interface unit, where the application program interface unit includes an application program interface corresponding to the first data, and the data channel cluster includes a first data channel corresponding to the first data. The gateway control module receives first data in a first format sent by the database, sends the first data to the data channel cluster, and the processing process of the gateway control module is that the application program interface unit receives the first data in the first format sent by the database through the first application program interface and sends the first data to a first data channel in the data channel cluster.
In an implementation, the gateway control module includes an application program interface unit. In order to distinguish different types of data, the application program interface unit includes application program interfaces for receiving different types of data (i.e. one application program interface in the application program interface unit corresponds to one type of data). Also, the processing is different for different types of data. Therefore, the data channel cluster includes data channels for processing different types of data (i.e., one data channel in the data channel cluster corresponds to one type of data). Therefore, one type of data corresponds to one application program interface in the application program interface unit and corresponds to one data channel in the data channel cluster (for example, a snapshot data channel corresponding to snapshot data, an alarm data channel corresponding to alarm data, a device data channel corresponding to device data, and the like). Correspondingly, for the first data, the application program interface unit includes a first application program interface corresponding to the first data, and the data channel cluster includes a first data channel corresponding to the first data. In this way, the application program interface unit in the gateway control module may receive the first data in the first format sent by the database through the first application program interface. The application interface unit may then send the first data to a first data channel of the cluster of data channels.
Step 202, the data channel cluster converts the first format of the first data into the second format to obtain the first data after format conversion, and sends the first data after format conversion to the application system.
The second format is a format for storing the first data in the application system.
In implementation, since the application system mainly focuses on the implementation of the video image application function in the initial development stage, the application system does not support the standard specification, that is, the format of the application system and the format of the first data stored in the database are not necessarily the same. Therefore, in order to ensure that the application system can store and process the first data, after receiving the first data, the data channel cluster may further convert the first format of the first data into the second format, so as to obtain the format-converted first data. The second format is a format for storing the first data in the application system. The data channel cluster may then send the format-converted first data to the application system. Accordingly, the application system may then receive the first data in the second format for storage and processing. Therefore, secondary development of the application system is not required to be performed by consuming manpower and material resources, and data interaction between the application system and the database is realized by arranging the data channel system between the application system and the database.
As an optional implementation manner, the data channel cluster includes a first data channel corresponding to the first data, and the first data channel includes a first data mapping table corresponding to the first data and a first data driver. The data channel cluster converts the first format of the first data into the second format to obtain the first data after format conversion, and sends the first data after format conversion to the application system in the following processing process.
Step one, a first data channel converts a first format of first data into a second format according to a first data mapping table to obtain the first data after format conversion.
In implementation, because format conversion processing of different types of data is different, storage manners of the different types of data in the application system are also different, and data drivers corresponding to the different storage manners are also different (for example, a Representational State Transfer (REST) interface corresponds to a hypertext Transfer Protocol (HTTP) client, a database corresponds to a database connection pool, and a message middleware corresponds to a consumer listener). Therefore, the data channel cluster includes data channels for processing different types of data. Wherein the data channel includes a data mapping table for format converting different types of data and a data driver for transmitting the different types of data. Correspondingly, for the first data, the data channel cluster includes a first data channel corresponding to the first data, and the first data channel includes a first data mapping table corresponding to the first data and a first data driver. Thus, after receiving the first data, the first data channel may convert the first format of the first data into the second format according to the first data mapping table, so as to obtain the format-converted first data.
And step two, the first data channel sends the first data after format conversion to an application system through a first data driver.
In implementation, after the first data channel obtains the format-converted first data, the format-converted first data may be sent to the application system through the first data driver. Accordingly, the application system may then receive the first data in the second format for storage and processing.
Step 203, the data channel cluster receives second data in a third format sent by the application system, converts the third format of the second data into a fourth format to obtain second data after format conversion, and sends the second data after format conversion to the gateway control module.
The third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database.
In an implementation, when an application system needs to upload a type of locally stored data (i.e., the second data) to the database, the application system may send the second data in the third format to the data channel cluster in the data channel system. And the third format is a format for storing the second data in the application system. Accordingly, the data channel cluster in the data channel system may receive the second data in the third format sent by the application system. Since the format of the application system and the second data stored in the database are not necessarily the same. Therefore, in order to ensure that the database can store the second data, the data channel cluster may further convert the third format of the second data into the fourth format, so as to obtain the format-converted second data. And the fourth format is a format for storing the second data in the database. The data channel cluster may then send the format-converted second data to the gateway control module.
As an alternative embodiment, the data channel cluster includes a second data channel corresponding to the second data, and the second data channel includes a second data mapping table and a second data driver corresponding to the second data. The data channel cluster receives second data in a third format sent by the application system, converts the third format of the second data into a fourth format to obtain second data after format conversion, and sends the second data after format conversion to the gateway control module in the following processing process.
Step one, a second data channel receives second data in a third format sent by an application system through a second data driver.
In implementation, because format conversion processing of different types of data is different, storage modes of different types of data in an application system are different, and data drives corresponding to different storage modes are also different (for example, a Representational State Transfer (REST) interface corresponds to a hypertext Transfer Protocol (HTTP) client, a database corresponds to a database connection pool, and a message middleware corresponds to a consumer listener). Therefore, the data channel cluster includes data channels for processing different types of data. Wherein the data channel includes a data mapping table for format converting different types of data and a data driver for receiving the different types of data. Correspondingly, for the second data, the data channel cluster includes a second data channel corresponding to the second data, and the second data channel includes a second data mapping table and a second data driver corresponding to the second data. In this way, the application system may send second data in a third format to a second data driver of a second data channel in the cluster of data channels. Accordingly, the second data channel may receive second data in a third format transmitted by the application system through the second data driver.
And step two, the second data channel converts the third format of the second data into the fourth format according to the second data mapping table to obtain the second data after format conversion, and sends the second data after format conversion to the gateway control module.
In implementation, after receiving the second data in the third format sent by the application system, the second data channel may convert the third format of the second data into the fourth format according to the second data mapping table, so as to obtain the second data after format conversion. The second data channel may then send the format-converted second data to the gateway control module.
And step 204, the gateway control module sends the second data after format conversion to a database.
In implementation, after receiving the second data after format conversion, the gateway control module may send the second data after format conversion to the database. Accordingly, the database may receive the second data in the fourth format to store the second data in the fourth format. Therefore, secondary development of the application system is not required to be performed by consuming manpower and material resources, and data interaction between the application system and the database is realized by arranging the data channel system between the application system and the database.
As an alternative embodiment, the gateway control module comprises a gateway unit. The processing process of the gateway control module sending the second data after format conversion to the database is that the gateway control module sends the second data after format conversion to the database through the gateway unit.
In an implementation, the gateway control module further comprises a gateway unit for sending data to the database. And after receiving the second data after the format conversion, the gateway control module can send the second data after the format conversion to the database through the gateway unit.
It should be noted that, steps 101 and 102 are processes in which the application system obtains first data from the database through the data channel system, and steps 103 and 104 are processes in which the application system uploads second data to the database through the data channel. The processing order of the two processes does not distinguish the sequence.
As an optional implementation manner, the data channel system further includes a channel management module, where the channel management module is configured to create a data channel cluster. As shown in fig. 3, the specific processing procedure is as follows:
step 301, the channel management module obtains a data driver and a data mapping table corresponding to each data.
In the implementation, the format conversion processing of different types of data is different, and the storage mode of different types of data in the application system is also different. Thus, different types of data correspond to different data drives and different data mapping tables (i.e., one type of data corresponds to one data drive and one data mapping table). The data channel system may be pre-stored with data drivers and data mapping tables corresponding to various types of data. The generation process of the data driver and the data mapping table corresponding to each type of data will be described in detail later. When the channel management module needs to create a data channel cluster corresponding to each type of data, the channel management module may obtain a data driver and a data mapping table corresponding to each type of data from a storage device in which the data driver and the data mapping table corresponding to each type of data are pre-stored in the data channel system. Optionally, the channel management module may create a data channel cluster when the data channel system is started and initialized, or create a data channel cluster according to an instruction of a user, which is not limited in this application.
Step 302, the channel management module creates a data channel corresponding to each data according to the data driver corresponding to each data and the data mapping table, and forms a data channel cluster.
In implementation, one type of data corresponds to one program interface in the application program interface unit, and one data channel corresponds to one data channel in the data channel cluster, so that after the channel management module obtains the data driver and the data mapping table corresponding to each type of data, for each type of data, the channel management module may create the data channel corresponding to the type of data according to the data driver and the data mapping table corresponding to the type of data, so as to obtain the data channels corresponding to each type of data, and form the data channel cluster. Optionally, the channel management module may further perform, according to an instruction of a user, an opening operation, a pause operation, a filter setting, a traffic monitoring setting, a destroy operation, a re-creation operation, a data channel cluster copy operation, and the like on a certain data channel in the data channel cluster. In addition, the channel management module can also provide a fusing protection strategy of the data channel when any one of the application system and the database is abnormally down.
In an alternative embodiment, the data path system includes a plurality of data path clusters, and the plurality of data path clusters includes a primary data path cluster and a backup data path cluster. The channel management module can
When the channel management module detects that the first data channel cluster fails, the second data channel cluster is switched to a main data channel cluster, the first data channel cluster is the main data channel cluster, and the second data channel cluster is a backup data channel cluster.
As an optional implementation, the data channel system further includes a metadata management module. The metadata management module may generate a drive and a data mapping table corresponding to each data according to the metadata of each data. As shown in fig. 4, the specific process is as follows.
Step 401, the metadata management module obtains metadata of each data.
In an implementation, metadata of each view data resource (i.e., each data) in an application system may be stored in advance in a data channel system, where the metadata is a data rule for describing an acquisition manner (e.g., REST interface, message middleware, database, big data warehouse, etc.) of each view data resource (e.g., face information, vehicle information, alarm information, acquisition device information, etc.) in the application system, and the metadata includes two parts, namely, data source (resources) and field mapping relationship (mappings).
Step 402, the metadata management module analyzes the metadata of each data to obtain the metadata of the data driver and the metadata of the data mapping table corresponding to each data.
In implementation, after the metadata management module obtains the metadata of each data, for the metadata of each data, the metadata management module may analyze the metadata of the data to obtain the data driver metadata and the data mapping table metadata corresponding to the data. The data driver metadata corresponds to a data source in the metadata, and the data mapping table metadata corresponds to a field mapping relation in the metadata.
Step 403, the metadata management module performs translation processing on the metadata of the data driver and the metadata of the data mapping table corresponding to each data to obtain the data driver and the data mapping table corresponding to each data, and stores the data driver and the data mapping table corresponding to each data.
In implementation, after the metadata management module obtains the metadata of the data driver and the metadata of the data mapping table corresponding to each data, for each data, the metadata management module may perform translation processing on the metadata of the data driver and the metadata of the data mapping table corresponding to the data to obtain the metadata of the data driver and the data mapping table corresponding to the data. Then, the metadata management module may store the data driver and the data mapping table corresponding to each data to the local. Optionally, the metadata management module may store the data driver and the data mapping table corresponding to each data into an ETCD (an open source supporting a highly available and highly concurrent distributed key-value storage system), so as to ensure that the data driver and the data mapping table for creating the data channel are not lost, thereby improving the stability of the channel system. In addition, multi-instance deployment can be carried out on a data channel corresponding to each type of data, and data idempotency among multiple instances can be ensured through the management of a data driver and a data mapping table of the ETCD.
Fig. 5A is a flowchart of an example of a data transmission method according to an embodiment of the present application. In this example, the snapshot data is taken as an example for introduction, the snapshot data corresponds to the snapshot program interface in the application program interface unit, corresponds to the snapshot data channel in the data channel cluster, and includes the snapshot data driver and the snapshot data mapping table in the snapshot data channel. As shown in fig. 5A, the process includes the following steps.
In step 511, the application program interface unit in the gateway control module receives snapshot data in the first format sent by the database through the snapshot application program interface. The first format is a format for storing snapshot data in the database.
Step 512, the application program interface unit in the gateway control module sends the snapshot data in the first format to the snapshot data channel in the data channel cluster.
Step 513, the snapshot data channel converts the first format of the snapshot data into the second format through the snapshot data mapping table, so as to obtain the snapshot data after format conversion. And the second format is a format for storing snapshot data in the application system.
And step 514, the snapshot data channel sends the snapshot data with the converted format to an application system through the snapshot data driver.
The processing procedures in steps 511 to 514 and the types of the processing procedures in steps 201 to 202 are not described herein again.
Fig. 5B is a flowchart of an example of a data transmission method according to an embodiment of the present application. In this example, the snapshot data is taken as an example for description, the snapshot data corresponds to a snapshot data channel in a data channel cluster, and the snapshot data channel includes a snapshot data driver and a snapshot data mapping table. As shown in fig. 5B, the process includes the following steps.
Step 521, the snapshot data channel in the data channel cluster receives the snapshot data in the second format sent by the application system through the snapshot data driver. And the second format is a format for storing snapshot data in the application system.
In step 522, the snapshot data channel converts the second format of the snapshot data into the first format through the snapshot data mapping table, so as to obtain the snapshot data after format conversion. The first format is a format for storing snapshot data in the database.
Step 523, the snapshot data channel sends the format-converted snapshot data to the gateway unit in the gateway control module.
Step 524, the gateway control module sends the snapshot data after format conversion to the database through the gateway unit.
The processing procedures of steps 521 to 524 and the types of the processing procedures of steps 203 to 204 are not described herein again.
The embodiment of the application provides a data transmission method. The method is applied to a data channel system. The data channel system comprises a gateway control module and a data channel cluster. The gateway control module is connected with the database, and the data channel cluster is connected with the application system. The gateway control module receives first data in a first format sent by the database and sends the first data to the data channel cluster. The first format is a format for storing first data in the database. The data channel cluster converts the first format of the first data into the second format to obtain the first data after format conversion, and sends the first data after format conversion to the application system. The second format is a format for storing the first data in the application system. And the data channel cluster receives second data in a third format sent by the application system, converts the third format of the second data into a fourth format to obtain second data after format conversion, and sends the second data after format conversion to the gateway control module. The third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database. And the gateway control module sends the second data after format conversion to a database. Therefore, secondary development of the application system is not required to be performed by consuming manpower and material resources, and the data channel system is arranged between the application system and the database, so that data interaction between the application system and the database is realized, and interaction between different application systems is realized by taking the database as a hub.
The embodiment of the present application further provides a data channel system, as shown in fig. 6, the data channel system includes a gateway control module 610 and a data channel cluster 620, the gateway control module 610 is connected to a database, and the data channel cluster 620 is connected to an application system;
the gateway control module 610 is configured to receive first data in a first format sent by the database, and send the first data to the data channel cluster 620; the first format is a format in which the first data is stored in the database;
the data channel cluster 620 is configured to convert the first format of the first data into a second format, obtain format-converted first data, and send the format-converted first data to the application system; the second format is a format for storing the first data in the application system;
the data channel cluster 620 is configured to receive second data in a third format sent by the application system, convert the third format of the second data into a fourth format, obtain second data after format conversion, and send the second data after format conversion to the gateway control module 610; the third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database;
the gateway control module 610 is configured to send the format-converted second data to the database.
As an optional implementation manner, the gateway control module 610 includes an application program interface unit, where the application program interface unit includes an application program interface corresponding to the first data, and the data channel cluster 620 includes a first data channel corresponding to the first data;
the application program interface unit is configured to receive first data in a first format sent by the database through the first application program interface, and send the first data to the first data channel in the data channel cluster 620.
As an optional implementation manner, the data channel cluster 620 includes a first data channel corresponding to the first data, where the first data channel includes a first data mapping table and a first data driver corresponding to the first data;
the first data channel is used for converting the first format of the first data into a second format according to the first data mapping table to obtain format-converted first data;
the first data channel is used for sending the first data after format conversion to the application system through the first data driver.
As an alternative embodiment, the data channel cluster 620 includes a second data channel corresponding to the second data, where the second data channel includes a second data mapping table and a second data driver corresponding to the second data;
the second data channel is used for receiving second data in a third format sent by the application system through the second data driver;
the second data channel is configured to convert the third format of the second data into a fourth format according to the second data mapping table, obtain second data with a converted format, and send the second data with the converted format to the gateway control module 610.
As an optional implementation, the gateway control module 610 includes a gateway unit;
the gateway control module 610 is configured to send the format-converted second data to the database through the gateway unit.
As an optional implementation, the data channel system further includes a channel management module 630;
the channel management module 630 is configured to obtain a data driver and a data mapping table corresponding to each data;
the channel management module 630 is configured to create a data channel corresponding to each data according to the data driver corresponding to each data and the data mapping table, so as to form the data channel cluster 620.
As an optional implementation manner, the data channel system includes a plurality of data channel clusters, where the plurality of data channel clusters include a main data channel cluster and a backup data channel cluster;
the channel management module 630 is configured to, when a failure of a first data channel cluster is detected, switch a second data channel cluster to a main data channel cluster, where the first data channel cluster is the main data channel cluster, and the second data channel cluster is a backup data channel cluster.
As an optional embodiment, the data channel system further includes a metadata management module 640;
the metadata management module 640 is configured to obtain metadata of each data;
the metadata management module 640 is configured to analyze the metadata of each data to obtain data driver metadata and data mapping table metadata corresponding to each data;
the metadata management module 640 is configured to perform translation processing on the data driver metadata and the data mapping table metadata corresponding to each piece of data to obtain a data driver and a data mapping table corresponding to each piece of data, and store the data driver and the data mapping table corresponding to each piece of data.
The embodiment of the application provides a data channel system. The data channel system comprises a gateway control module and a data channel cluster. The gateway control module is connected with the database, and the data channel cluster is connected with the application system. The gateway control module receives first data in a first format sent by the database and sends the first data to the data channel cluster. The first format is a format for storing first data in the database. The data channel cluster converts the first format of the first data into the second format to obtain the first data after format conversion, and sends the first data after format conversion to the application system. The second format is a format for storing the first data in the application system. And the data channel cluster receives second data in a third format sent by the application system, converts the third format of the second data into a fourth format to obtain second data after format conversion, and sends the second data after format conversion to the gateway control module. The third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database. And the gateway control module sends the second data after format conversion to a database. Therefore, secondary development of the application system is not required to be performed by consuming manpower and material resources, and the data channel system is arranged between the application system and the database, so that data interaction between the application system and the database is realized, and interaction between different application systems is realized by taking the database as a hub.
In one embodiment, a computer device is provided, as shown in fig. 6, and includes a memory and a processor, where the memory stores a computer program that can be executed on the processor, and the processor implements the data transmission method when executing the computer program.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when being executed by a processor, carries out the above-mentioned method of data transmission.
It will be understood by those of ordinary skill in the art that all or a portion of the processes of the methods of the embodiments described above may be implemented by a computer program that may be stored on a non-volatile computer-readable storage medium, which when executed, may include the processes of the embodiments of the methods described above, wherein any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (11)
1. A data transmission method is applied to a data channel system, the data channel system comprises a gateway control module and a data channel cluster, the gateway control module is connected with a database, the data channel cluster is connected with an application system, and the method comprises the following steps:
the gateway control module receives first data in a first format sent by the database and sends the first data to the data channel cluster; the first format is a format in which the first data is stored in the database;
the data channel cluster converts the first format of the first data into a second format to obtain first data after format conversion, and sends the first data after format conversion to the application system; the second format is a format for storing the first data in the application system;
the data channel cluster receives second data in a third format sent by the application system, converts the third format of the second data into a fourth format to obtain second data with converted formats, and sends the second data with converted formats to the gateway control module; the third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database;
and the gateway control module sends the second data after format conversion to the database.
2. The method of claim 1, wherein the gateway control module comprises an application program interface unit, the application program interface unit comprises an application program interface corresponding to the first data, and the data channel cluster comprises a first data channel corresponding to the first data;
the gateway control module receives first data in a first format sent by the database, and sends the first data to the data channel cluster, and the method comprises the following steps:
and the application program interface unit receives first data in a first format sent by the database through the first application program interface and sends the first data to the first data channel in the data channel cluster.
3. The method of claim 1 or 2, wherein the data channel cluster includes a first data channel corresponding to the first data, the first data channel including a first data mapping table and a first data driver corresponding to the first data;
the data channel cluster converts the first format of the first data into a second format to obtain first data after format conversion, and sends the first data after format conversion to the application system, and the method comprises the following steps:
the first data channel converts the first format of the first data into a second format according to the first data mapping table to obtain first data after format conversion;
and the first data channel sends the format-converted first data to the application system through the first data driver.
4. The method of claim 1, wherein the data channel cluster contains a second data channel corresponding to the second data, the second data channel comprising a second data mapping table and a second data driver corresponding to the second data;
the data channel cluster receives second data in a third format sent by the application system, converts the third format of the second data into a fourth format to obtain second data after format conversion, and sends the second data after format conversion to the gateway control module, and the method includes:
the second data channel receives second data in a third format sent by the application system through the second data driver;
and the second data channel converts the third format of the second data into a fourth format according to the second data mapping table to obtain second data with converted formats, and sends the second data with converted formats to the gateway control module.
5. The method of claim 1 or 4, wherein the gateway control module comprises a gateway unit;
the gateway control module sends the second data after format conversion to the database, and the method comprises the following steps:
and the gateway control module sends the second data after format conversion to the database through the gateway unit.
6. The method of claim 1, wherein the data channel system further comprises a channel management module, the method further comprising:
the channel management module acquires a data driver and a data mapping table corresponding to each data;
and the channel management module creates a data channel corresponding to each data according to the data driver corresponding to each data and the data mapping table to form the data channel cluster.
7. The method of claim 6, wherein the data tunnel system comprises a plurality of data tunnel clusters, the plurality of data tunnel clusters comprising a primary data tunnel cluster and a backup data tunnel cluster, the method further comprising:
when the channel management module detects that a first data channel cluster fails, a second data channel cluster is switched to a main data channel cluster, the first data channel cluster is a main data channel cluster, and the second data channel cluster is a backup data channel cluster.
8. The method of claim 1, wherein the data channel system further comprises a metadata management module, the method further comprising:
the metadata management module acquires metadata of each data;
the metadata management module analyzes the metadata of each data to obtain the metadata of a data driver and the metadata of a data mapping table corresponding to each data;
and the metadata management module translates the metadata of the data driver and the metadata of the data mapping table corresponding to each data to obtain the data driver and the data mapping table corresponding to each data, and stores the data driver and the data mapping table corresponding to each data.
9. A data channel system is characterized in that the data channel system comprises a gateway control module and a data channel cluster, wherein the gateway control module is connected with a database, and the data channel cluster is connected with an application system;
the gateway control module is used for receiving first data in a first format sent by the database and sending the first data to the data channel cluster; the first format is a format in which the first data is stored in the database;
the data channel cluster is used for converting the first format of the first data into a second format to obtain first data after format conversion, and sending the first data after format conversion to the application system; the second format is a format for storing the first data in the application system;
the data channel cluster is used for receiving second data in a third format sent by the application system, converting the third format of the second data into a fourth format to obtain second data after format conversion, and sending the second data after format conversion to the gateway control module; the third format is a format for storing the second data in the application system, and the fourth format is a format for storing the second data in the database;
and the gateway control module is used for sending the second data after the format conversion to the database.
10. A computer device comprising a memory and a processor, the memory having stored thereon a computer program operable on the processor, wherein the processor, when executing the computer program, performs the steps of the method of any of claims 1 to 8.
11. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 8.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010174731.3A CN111405035A (en) | 2020-03-13 | 2020-03-13 | Data transmission method and data channel system |
| PCT/CN2020/119709 WO2021179589A1 (en) | 2020-03-13 | 2020-09-30 | Data transmission method and data channel system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010174731.3A CN111405035A (en) | 2020-03-13 | 2020-03-13 | Data transmission method and data channel system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111405035A true CN111405035A (en) | 2020-07-10 |
Family
ID=71432507
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010174731.3A Pending CN111405035A (en) | 2020-03-13 | 2020-03-13 | Data transmission method and data channel system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111405035A (en) |
| WO (1) | WO2021179589A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021179589A1 (en) * | 2020-03-13 | 2021-09-16 | 北京旷视科技有限公司 | Data transmission method and data channel system |
| CN115037794A (en) * | 2022-06-07 | 2022-09-09 | 抖音视界(北京)有限公司 | Data acquisition method and device, computer equipment and readable storage medium |
| CN115103224A (en) * | 2022-06-07 | 2022-09-23 | 慧之安信息技术股份有限公司 | Intelligent video analysis method supporting GAT 1400 protocol |
| CN115103021A (en) * | 2022-05-18 | 2022-09-23 | 慧之安信息技术股份有限公司 | Gateway access management method based on GAT1400 protocol |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1631002A (en) * | 2000-07-18 | 2005-06-22 | 皇家菲利浦电子有限公司 | Mobile apparatus enabling inter-network communication |
| CN105094707A (en) * | 2015-08-18 | 2015-11-25 | 华为技术有限公司 | Method and device for storing and reading data |
| CN105743709A (en) * | 2016-04-05 | 2016-07-06 | 浪潮电子信息产业股份有限公司 | Management device, complete cabinet management system and management method |
| CN110162571A (en) * | 2019-04-26 | 2019-08-23 | 厦门市美亚柏科信息股份有限公司 | A kind of system, method, storage medium that data among heterogeneous databases synchronize |
| US20190370698A1 (en) * | 2012-05-09 | 2019-12-05 | Nasdaq Technology Ab | Methods and arrangements for an automated exchange system |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1217277C (en) * | 2002-04-01 | 2005-08-31 | 上海网上乐园信息技术有限公司 | Compasite distributed information processing system and implementation method thereof |
| CN101754221B (en) * | 2008-12-19 | 2012-07-04 | 中国移动通信集团山东有限公司 | Data transmission method between heterogeneous systems and data transmission system |
| US10587579B2 (en) * | 2017-05-19 | 2020-03-10 | Vmware, Inc. | Varying encryption level of traffic through network tunnels |
| CN109525539B (en) * | 2017-09-20 | 2021-04-02 | 中移（杭州）信息技术有限公司 | Capability calling method, call receipt method, information processing method and system |
| CN111405035A (en) * | 2020-03-13 | 2020-07-10 | 北京旷视科技有限公司 | Data transmission method and data channel system |
-
2020
- 2020-03-13 CN CN202010174731.3A patent/CN111405035A/en active Pending
- 2020-09-30 WO PCT/CN2020/119709 patent/WO2021179589A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1631002A (en) * | 2000-07-18 | 2005-06-22 | 皇家菲利浦电子有限公司 | Mobile apparatus enabling inter-network communication |
| US20190370698A1 (en) * | 2012-05-09 | 2019-12-05 | Nasdaq Technology Ab | Methods and arrangements for an automated exchange system |
| CN105094707A (en) * | 2015-08-18 | 2015-11-25 | 华为技术有限公司 | Method and device for storing and reading data |
| CN105743709A (en) * | 2016-04-05 | 2016-07-06 | 浪潮电子信息产业股份有限公司 | Management device, complete cabinet management system and management method |
| CN110162571A (en) * | 2019-04-26 | 2019-08-23 | 厦门市美亚柏科信息股份有限公司 | A kind of system, method, storage medium that data among heterogeneous databases synchronize |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021179589A1 (en) * | 2020-03-13 | 2021-09-16 | 北京旷视科技有限公司 | Data transmission method and data channel system |
| CN115103021A (en) * | 2022-05-18 | 2022-09-23 | 慧之安信息技术股份有限公司 | Gateway access management method based on GAT1400 protocol |
| CN115037794A (en) * | 2022-06-07 | 2022-09-09 | 抖音视界(北京)有限公司 | Data acquisition method and device, computer equipment and readable storage medium |
| CN115103224A (en) * | 2022-06-07 | 2022-09-23 | 慧之安信息技术股份有限公司 | Intelligent video analysis method supporting GAT 1400 protocol |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021179589A1 (en) | 2021-09-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111405035A (en) | Data transmission method and data channel system | |
| CN109542865B (en) | Method, device, system and medium for synchronizing configuration files of distributed cluster system | |
| EP3364310A1 (en) | Data processing method and device | |
| US20140078930A1 (en) | Method and network management system for reconfiguring a data network node | |
| CN105159795A (en) | Data synchronization method, apparatus and system | |
| CN109558199B (en) | Information pushing method, device, equipment and storage medium | |
| US11837016B2 (en) | Intelligent analysis system, method and device | |
| CN110750379B (en) | ETCD cluster recovery method, system, equipment and computer medium | |
| CN106941522B (en) | Lightweight distributed computing platform and data processing method thereof | |
| US20240184675A1 (en) | Data synchronization method for host machine and backup machine of station application server | |
| EP3171565B1 (en) | Methods, devices and system for netconf hello packets interaction | |
| CN111601299B (en) | Information association backfill system under 5G framework | |
| CN113259425A (en) | 61850 standard-based data conversion method, transformer substation gateway and storage medium | |
| US20140032744A1 (en) | Method of comparing outputs in a plurality of information systems | |
| CN109257403B (en) | Data storage method and device and distributed storage system | |
| CN112035062A (en) | Migration method of local storage of cloud computing, computer equipment and storage medium | |
| US9870249B2 (en) | Virtual computer system, method, and non-transitory computer readable medium | |
| CN106209407B (en) | Method and equipment for session backup | |
| US10931363B2 (en) | Relay device and non-transitory computer readable medium | |
| CN116032796A (en) | Connection state detection method and related equipment | |
| CN114327563A (en) | Data synchronization method, device, system, storage medium and computer system | |
| CN112765279A (en) | Database synchronization method and system | |
| CN115993979A (en) | Configuration conversion method, smooth upgrading method, device, equipment and storage medium | |
| CN110968557B (en) | Data processing method and device in distributed file system and electronic equipment | |
| CN102281159A (en) | Recovery method for cluster system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200710 |
|
| RJ01 | Rejection of invention patent application after publication |