CN110618880B - Cross-system data transmission system and method - Google Patents
Cross-system data transmission system and method Download PDFInfo
- Publication number
- CN110618880B CN110618880B CN201910886758.2A CN201910886758A CN110618880B CN 110618880 B CN110618880 B CN 110618880B CN 201910886758 A CN201910886758 A CN 201910886758A CN 110618880 B CN110618880 B CN 110618880B
- Authority
- CN
- China
- Prior art keywords
- data
- source
- target system
- target
- processing unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer And Data Communications (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a cross-system data transmission system and a method, wherein the system comprises: the device comprises a preprocessing module, a data processing module and a data sending module; the system comprises a pre-processing module, a processing unit and a data processing module, wherein the pre-processing module is used for receiving source data downloaded by a source system, distributing the source data to the processing unit and monitoring performance data of the processing unit; adjusting the source data distributed to the processing units according to the monitored performance data of the processing units; the data processing module is arranged in the processing unit and used for acquiring a target system data template from the target system template set according to the mapping rule and the source data; processing the source data in the processing unit into target data according to the target system data template and the mapping rule; and the data sending module is used for transmitting the target data to the target system. The invention monitors the performance data of the processing unit, and adjusts the data distribution in real time for the processing unit with reduced processing performance, so that the processing unit can process data efficiently, and the efficiency and performance of data transmission are improved.
Description
Technical Field
The present invention relates to the field of data transmission technologies, and in particular, to a cross-system data transmission system and method.
Background
In the existing data transmission method, overall transmission or fixed fragment transmission is mostly adopted, and when the transmission quantity of a certain fragment is increased rapidly, the performance of the fragment is reduced and the fragment is not easy to mediate.
In the existing data transmission method, a fixed agreed template is used between a source system and a target system, all transaction data are transmitted according to the existing template, and when a new target system and different data acquisition results appear, a data transmission flow needs to be set up again.
Disclosure of Invention
The embodiment of the invention provides a cross-system data transmission system, which monitors performance data of a processing unit through a front-end processing module and adjusts data distribution in real time, and comprises the following components: the device comprises a preprocessing module, a data processing module and a data sending module;
the system comprises a pre-processing module, a processing unit and a data processing module, wherein the pre-processing module is used for receiving source data downloaded by a source system, distributing the source data to the processing unit and monitoring performance data of the processing unit; adjusting the source data distributed to the processing unit according to the monitored performance data of the processing unit;
the data processing module is arranged in the processing unit and used for acquiring a target system data template from the target system template set according to the mapping rule and the source data; processing the source data in the processing unit into target data according to the target system data template and the mapping rule;
and the data sending module is used for transmitting the target data to the target system.
In one embodiment, the cross-system data transfer system according to the embodiment of the present invention further includes:
the template generation module is used for acquiring the information of the newly added target system when the target system is newly added; and generating a data template of the newly added target system according to the information of the newly added target system, and adding the data template to the template set of the target system.
The embodiment of the invention also provides a cross-system data transmission method, which comprises the following steps:
receiving source data downloaded by a source system, distributing the source data to a processing unit, and monitoring performance data of the processing unit; adjusting the source data distributed to the processing units according to the monitored performance data of the processing units;
acquiring a target system data template from the target system template set according to the mapping rule and the source data; processing the source data in the processing unit into target data according to the target system data template and the mapping rule;
the target data is transferred to the target system.
In one embodiment, the cross-system data transfer method provided in the embodiment of the present invention further includes:
when a target system is newly added, acquiring information of the newly added target system; and generating a data template of the newly added target system according to the information of the newly added target system, and adding the data template to the template set of the target system.
The embodiment of the invention also provides computer equipment which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor executes the computer program to realize the cross-system data transmission method.
An embodiment of the present invention further provides a computer-readable storage medium, where a computer program for implementing the cross-system data transmission method is stored in the computer-readable storage medium.
According to the cross-system data transmission system and method provided by the embodiment of the invention, aiming at the transmission of cross-system data, the performance data of the processing unit is monitored, and the data distribution is adjusted in real time for the processing unit with reduced processing performance, so that the processing unit can efficiently process data, and the efficiency and performance of data transmission are improved.
Furthermore, the embodiment of the invention does not limit the variety increase of the target system any more, and can expand the transaction template set according to different entity transactions and newly added target systems, thereby meeting the data requirements of diversified target systems and enhancing the expandability of data transmission.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
fig. 1 is a schematic diagram of a cross-system data transfer system according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of a data transfer process of a cross-system data transfer system according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of a source data format received by the cross-system data delivery system according to an embodiment of the present invention.
Fig. 4 is a schematic diagram of a source data distribution process of a cross-system data delivery system according to an embodiment of the present invention.
Fig. 5 is a schematic diagram of a data processing process of a cross-system data transmission system according to an embodiment of the present invention.
Fig. 6 is a schematic diagram of a target data format of a cross-system data transfer system according to an embodiment of the present invention.
Fig. 7 is a schematic diagram of a cross-system data transfer method according to an embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
As shown in fig. 1 and fig. 2, which are a schematic diagram of a cross-system data transfer system according to an embodiment of the present invention and a schematic diagram of a data transfer process of a cross-system data transfer system according to an embodiment of the present invention, an embodiment of the present invention provides a cross-system data transfer system, which monitors performance data of a processing unit through a pre-processing module and adjusts data allocation in real time, and the system includes: the device comprises a preprocessing module, a data processing module and a data sending module;
the pre-processing module 101 is configured to receive source data downloaded by a source system, distribute the source data to a processing unit, and monitor performance data of the processing unit; adjusting the source data distributed to the processing units according to the monitored performance data of the processing units;
the data processing module 102 is arranged in the processing unit and is used for acquiring a target system data template from the target system template set according to the mapping rule and the source data; processing the source data in the processing unit into target data according to the target system data template and the mapping rule;
and the data sending module 103 is used for transmitting the target data to the target system.
In one embodiment, the cross-system data transfer system according to the embodiment of the present invention further includes:
the template generation module is used for acquiring the information of the newly added target system when the target system is newly added; and generating a data template of the newly added target system according to the information of the newly added target system, and adding the data template to the template set of the target system.
According to the cross-system data transmission system provided by the embodiment of the invention, aiming at the cross-system data transmission, the performance data of the processing unit is monitored, and the data distribution is adjusted in real time for the processing unit with reduced processing performance, so that the processing unit can efficiently process data, and the efficiency and the performance of data transmission are improved. Furthermore, the embodiment of the invention does not limit the increase of the types of the source system and the target system, and can expand the transaction template set according to different entity transactions and newly added target systems, thereby meeting the data requirements of diversified target systems and enhancing the expandability of data transmission.
The cross-system data transmission system provided by the embodiment of the invention supports data transmission pointing to multiple systems, and adjusts the processing capacity of data in a processing unit in real time so as to achieve optimal performance of the processing unit; as shown in fig. 1 and fig. 2, which are a schematic diagram of a cross-system data transfer system according to an embodiment of the present invention and a schematic diagram of a data transfer process of a cross-system data transfer system according to an embodiment of the present invention, in a specific implementation of the cross-system data transfer system according to an embodiment of the present invention, source data is sent out by a source system S, and is processed by the cross-system data transfer system according to an embodiment of the present invention, and then is transferred to a target system D1 — a target system Dn, where the cross-system data transfer system according to an embodiment of the present invention may include: the device comprises a preprocessing module, a data processing module and a data sending module;
the pre-processing module is used for receiving source data downloaded by a source system, distributing the source data to the processing unit and monitoring performance data of the processing unit; adjusting the source data distributed to the processing units according to the monitored performance data of the processing units; in an embodiment, the pre-processing module may be connected to a source system, and the source system downloads source data to the pre-processing module; the method comprises the following steps that a front-end processing module receives source data downloaded by a source system and distributes the source data to a processing unit; in the embodiment, the processing of the source data is completed inside the processing unit, and the system of the embodiment of the invention comprises a plurality of processing units 1-n; when the processing unit processes the source data, the performance data can be changed; for example, when the amount of source data allocated to the processing unit is too large, the processing unit may generate a problem of slow processing or performance degradation, so that the performance data is changed, and the fragmented data transfer performance is degraded.
The data processing module is arranged in the processing unit and used for acquiring a target system data template from a target system template set according to the mapping rule and the source data; processing the source data in the processing unit into target data according to the target system data template and the mapping rule; in an embodiment, the data processing module is arranged in each processing unit and performs specific refinement processing on the source data distributed to the processing unit; acquiring a target system data template from a target system template set according to the mapping rule and the source data; in the embodiment, a plurality of target system data templates are stored in the target system template set, and when a corresponding target system transmits data, the target system template corresponding to the target system can be obtained according to source data and a mapping rule; each target system corresponds to a unique target system template; and after the target system data template is obtained, processing the source data in the processing unit into target data according to the target system data template and the mapping rule.
The data sending module is used for transmitting the target data to the target system. In an embodiment, the target data processed by the data processing module may be transferred to the target system through the message bus MQ, so as to complete data transmission.
In order to solve the problem that a data transfer flow needs to be set up again when a new target system and different data acquisition results appear, the cross-system data transfer system of the embodiment of the invention may further include a template generation module for acquiring information of the newly added target system when one to multiple target systems are newly added; and generating a data template of the newly added target system according to the information of the newly added target system, and adding the data template to the template set of the target system. In the embodiment, each time a new target system appears, the template generation module subscribes data to the source system and the new target system, acquires information of the new target system, generates a new target system data template according to the acquired information of the new target system, and adds the new target system data template to the target system template set. Through the template generation module, the cross-system data transmission system can generate diversified templates for a newly-added target system in time according to the requirements of the target system when the newly-added target system or different entities trade, the increase of the types of a source system and the target system is not limited, a trading template set can be expanded according to different entity trades and the newly-added target system, the data requirements of the diversified target system are met, and the expandability of data transmission is enhanced.
As shown in fig. 3, which is a schematic diagram of a source data format received by a cross-system data delivery system according to an embodiment of the present invention, in an implementation of source data downloaded by a source system received by the embodiment of the present invention, the source data may include a source packet header and a source packet body; the source packet header includes at least: source system information, target system information, type information, fragmentation information and other data packet header information; a source packet body including at least: the source system transaction code information, transaction content information 1, transaction content information 2, … …, transaction content information n-1 and transaction content information n; the foregoing pre-processing module, in one embodiment, may include: the system comprises a plurality of preprocessing modules and a monitoring module; the preprocessing module is used for distributing the source data to the processing units according to the fragmentation information of the source data and the fragmentation rule; the monitoring module is used for monitoring the performance data of the processing unit and transmitting the performance data to the preprocessing module; and the preprocessing module is also used for adjusting the source data distributed to the processing unit according to the performance data monitored by the monitoring module. In an embodiment, the pre-processing module may allocate the source data to different pre-processing modules 1 to n according to the type information of the source packet header; as shown in fig. 4, which is a schematic diagram illustrating a source data distribution process of a cross-system data transmission system according to an embodiment of the present invention, a pre-processing module identifies type information dateHead 1-dateHead n of a source packet header, where a pre-processing module 1 corresponds to the type information dateHead 1 of the source packet header, a pre-processing module 2 corresponds to the type information dateHead 2 of the source packet header, and so on, the pre-processing module n corresponds to the type information dateHead n of the source packet header; distributing the source data comprising different types of information to corresponding preprocessing modules for processing through the corresponding relation; the preprocessing module 1 processes all source data with a source packet header type of dateHead 1, the preprocessing module 2 processes all source data with a source packet header type of dateHead 2, and so on, and the preprocessing module n processes all source data with a source packet header type of dateHead n; and each preprocessing module n processes data with all types of data being n. In an embodiment, the preprocessing module comprises a plurality of processing units 1-n, wherein the processing units 1-n are configured in number, the preprocessing module identifies fragment information of the allocated source data, and allocates the source data to the processing units 1-n according to a fragment rule; the fragment information refers to unique ID number data of the source data; the fragmentation rule can be set according to the user requirements and the performance of the processing unit; in an embodiment, the fragmentation rule may be an adaptive fragmentation rule that is uniformly distributed, and the adaptive fragmentation rule is mainly capable of performing adaptive adjustment according to the monitored data fed back by the monitoring module. In an embodiment, the monitoring module is connected to each processing unit, and configured to monitor performance data of each processing unit and transmit the performance data to the preprocessing module; in an embodiment, the preprocessing module is further configured to receive performance data of the processing unit sent by the monitoring module, and adjust source data allocated to the processing unit according to the performance data monitored by the monitoring module, so that the processing unit is stably maintained in a high performance state.
The performance data of the processing unit intercepted by the aforementioned interception module may include, in an embodiment: performance parameters and their alarm thresholds; the adjusting the data allocated to the processing unit according to the monitored performance data of the processing unit includes: when the performance parameters of the processing unit monitored by the monitoring module reach the alarm threshold value, triggering alarm information and transmitting the alarm information to the preprocessing module; and the preprocessing module adjusts the transmission interval of the source data distributed to the processing unit according to the alarm information. In an embodiment, the performance parameter of the processing unit may be set as the usage condition of the CPU, and the alarm threshold of the performance parameter may be set as a percentage of the rating of the performance parameter; the aforementioned preprocessing module adjusts data allocated to the processing unit according to the monitored performance data of the processing unit, and includes: when the performance parameters of the processing unit monitored by the monitoring module reach a set performance parameter alarm threshold, triggering alarm information and transmitting the alarm information to the preprocessing module; and the preprocessing module receives the alarm information and adjusts the transmission interval of the source data distributed to the processing unit according to the alarm information, so that the transmission quantity of the fragments is adjusted, and the performance of the processing unit is adjusted. For example, in an embodiment of the present invention, one preprocessing module includes 4 processing units, ID numbers of fragment information of source data belonging to a current preprocessing module are randomly distributed to 00 to 35 (36 bits in total), an average is initially set, monitoring is performed according to a performance parameter P, and an alarm threshold is set to be a threshold of 50% (parameterized and adjustable with practice), specifically as follows:
the processing unit 1 processes the data packet with ID number modulo 4 equal to 0 (i.e. ID number 00,04,08,12,16,20,24,28,32),
the processing unit 2 processes the data packet with ID number modulo 4 equal to 1 (i.e. ID number 01,05,09,13,17,21,25,29,33),
the processing unit 3 processes the data packet with ID number modulo 4 equal to 2 (i.e. ID number 02,06,10,14,18,22,26,30,34),
the processing unit 4 processes the data packet with ID number modulo 4 equal to 3 (i.e. ID number 03,07,11,15,19,23,27,31, 35);
if the number of the processing units exceeds 4, the same can be done according to the format; the monitoring module monitors the data magnitude and the processing performance parameters of the 4 processing units in real time, and when the processing of a certain processing unit is slow or the performance parameters are reduced to reach a set alarm threshold value, an alarm is triggered; for example, monitoring performance parameter P, when P1> threshold of processing unit 1, then adaptively adjusting a part of packets of processing unit 1 to the best performing processing unit among other 3 processing units, and if the alarm threshold is still exceeded, then continuing to perform mediation according to the rule.
As shown in fig. 5, in an embodiment of the present invention, a schematic diagram of a data processing process of a cross-system data transmission system, where the data processing module obtains a target system data template from a target system template set according to a mapping rule and source data, and includes: reading target system information in source data; and acquiring a target system data template from the target system template set according to the mapping rule and the target system information. The aforementioned target system information in the source data may be obtained in a source packet header of the source data. After the target system information is acquired, searching a target system data template in a target system template set according to a mapping rule; the mapping rule comprises: the unique mapping of the source data target system information and the target system data template, namely the source data target system information and the target system data template have a corresponding relation; in an embodiment, the mapping rule may further include: the unique mapping of the source system transaction code information and the target system transaction code information is that the source system transaction code information and the target system transaction code information have a corresponding relationship; in an embodiment, the mapping rule may further include: aiming at the unique corresponding target format of the target system transaction code, namely the target system transaction code and the target format have a corresponding relation; in an embodiment, after obtaining the target system information, searching for the target system data template in the target system template set according to the mapping rule, and obtaining the target system data template required by data transfer may include: and acquiring the target system data template from the target system template set by utilizing the corresponding relation between the source data target system information and the target system data template in the mapping rule according to the source data target system information.
As shown in fig. 5, a schematic diagram of a data processing process of a cross-system data transmission system according to an embodiment of the present invention and fig. 6, a schematic diagram of a target data format of a cross-system data transmission system according to an embodiment of the present invention, where the data processing module processes source data in a processing unit into target data according to a target system data template and a mapping rule, in an embodiment, the data processing module may include: unpacking a source packet header and a source packet of source data, and removing the source packet header; processing the source data packet body into a target system transaction data packet according to a target system data template according to the mapping rule; the identification data generated when the processing unit processes the source data is used as a target data packet header; and packaging the target system transaction data packet as a target data packet body with the target data packet header to generate target data. In an embodiment, the aforementioned mapping rule may include: unique mapping of source system transaction code information and target system transaction code information and a unique corresponding target format for the target system transaction code; processing the source data packet body into a target system transaction data packet according to a target system data template according to the mapping rule, wherein the processing comprises the following steps: acquiring source system transaction code information from source data, and mapping the source system transaction code information to unique target system transaction code information according to the corresponding relation between the source system transaction code information and the target system transaction code information in the mapping rule; acquiring a unique corresponding target format according to the target system transaction code information through the corresponding relation between the target system transaction code and the target format in the mapping rule, and determining the target format of a data packet required by final data transmission; and processing the source data packet body into a target system transaction data packet according to the target format. The processing unit generates identification data during the processing process, takes the generated identification data as a header of target data, takes the generated target system transaction data packet as a target data packet body, and packages the target data packet with the target data header to generate target data.
The processing unit generates the identification data during the processing, and in an embodiment, the processing may include: one or any combination of source system information, target system information, running water information and target system transaction code information. The running water information comprises a unique running water number.
The source packet body may include: source system transaction code information; the aforementioned mapping rule may include: unique mapping of source system transaction code information to target system transaction code information. The mapping rule can also comprise a unique corresponding relation between source data target system information and a target system data template, the data processing module acquires the target system data template from the target system template set through the unique corresponding relation, and the unique mapping can realize quick and accurate retrieval and acquisition of the target system data template. The aforementioned mapping rule may further include a uniquely corresponding target format for the target system transaction code.
As shown in fig. 5 and 6, in one example of the embodiment of the present invention, after the source data is distributed to the processing units, the data processing module identifies the target system information in the source packet header, and identifies the corresponding target system 1, target system 2, … …, and target system n.
In the target system 1, acquiring a target system 1 data template corresponding to the source data packet from a target system template set according to the corresponding relation between the source data target system information and the target system data template in the mapping rule; unpacking a source packet header and a source packet body of source data, and removing the source packet header; acquiring source system transaction code information of a source data packet body 1-a source data packet body n from source data, mapping the source system transaction code information to unique target system transaction code information according to the corresponding relation between the source system transaction code information and the target system transaction code information in the mapping rule, and acquiring a unique corresponding target format 1-a target format n according to the target system transaction code information through the corresponding relation between the target system transaction code and the target format in the mapping rule; processing the source packet body 1-the source packet body n into a transaction data packet of the target system 1 according to the target format 1-the target format n; the transaction data packet of the target system 1 includes: destination format 1 packet-destination format n packet.
Similarly, in the target system 2, according to the corresponding relationship between the source data target system information in the mapping rule and the target system data template, the target system 2 data template corresponding to the source data packet is obtained from the target system template set; unpacking a source packet header and a source packet body of source data, and removing the source packet header; acquiring source system transaction code information of a source data packet body 1-a source data packet body n from source data, mapping the source system transaction code information to unique target system transaction code information according to the corresponding relation between the source system transaction code information and the target system transaction code information in the mapping rule, and acquiring a unique corresponding target format 1-a target format n according to the target system transaction code information through the corresponding relation between the target system transaction code and the target format in the mapping rule; processing the source packet body 1-the source packet body n into a target system 2 transaction data packet according to the target format 1-the target format n; the target system 2 transaction data packet includes: destination format 1 packet-destination format n packet.
Similar to the processing procedure of the target system 1 or the target system 2, the cross-system data transfer system according to the embodiment of the present invention may support more target systems; the processing procedure of the source data in each target system is similar to that of the target system 1 or the target system 2, and is not described herein again.
The identification data generated when the processing unit processes the source data is used as a target data packet header; and packaging the target system transaction data packet as a target data packet body with the target data packet header to generate target data.
The aforementioned information of the new target system of the template generating module may, in an embodiment, include: adding a processing tool for converting the interface type and the entity type of the target system into a target format; the generating of the data template of the newly added target system according to the information of the newly added target system includes: acquiring an interface type of a newly added target system, and packaging the interface type in an entity class; and converting the entity class and the entity class into a processing tool with a target format to form a newly-added target system data template. In the embodiment, each target system template existing in the target system template set includes a plurality of different target formats 1 to n, so the acquired newly added target system data template also needs to include a plurality of different target formats 1 to n, and the acquired interface type of the newly added target system also includes a plurality of interface types 1 to n; when a plurality of newly added target systems are adopted, each interface type is packaged in the entity type according to different interface types of the newly added target systems, and the obtained interface types and the entity types are converted into the processing tools and the entity types in the target format to form a newly added target system data template which is added into a target system template set. In one example, the set of target system templates includes:
{
{ target system data template 1 target format 1, target system data template 1 target format 2, … …, target system data template 1 target format n },
{ target system data template 2 target format 1, target system data template 2 target format 2, … …, target system data template 2 target format n },
……，
{ target system data template n target format 1, target system data template 1 target format 2, … …, target system data template n target format n },
}
the embodiment of the invention can support a plurality of target systems, a set of target system data templates are added when one target system is added, the source data in the processing unit determines which target format is used according to the mapping rule (for example, different target formats are judged according to different transaction codes), and then the target data in the target format is generated for cross-system data transmission.
The embodiment of the invention also provides a cross-system data transmission method, which is described in the following embodiment. Because the principle of solving the problem of the method is similar to that of a cross-system data transmission system, the implementation of the method can be referred to the implementation of the cross-system data transmission system, and repeated details are not repeated.
As shown in fig. 7, a schematic diagram of a cross-system data transfer method according to an embodiment of the present invention is provided, where the embodiment of the present invention further provides a cross-system data transfer method, including:
step 701: receiving source data downloaded by a source system, distributing the source data to a processing unit, and monitoring performance data of the processing unit; adjusting the source data distributed to the processing units according to the monitored performance data of the processing units;
step 701: acquiring a target system data template from a target system template set according to the mapping rule and the source data; processing the source data in the processing unit into target data according to the target system data template and the mapping rule;
step 703: the target data is transferred to the target system.
In one embodiment, the method further comprises:
when a target system is newly added, acquiring information of the newly added target system; and generating a data template of the newly added target system according to the information of the newly added target system, and adding the data template to the template set of the target system.
In one embodiment, the source data includes: a source packet header and a source packet body; the source packet header at least comprises: slicing information;
a pre-processing module comprising: the system comprises a preprocessing module and a monitoring module;
the preprocessing module is used for distributing the source data to the processing units according to the fragmentation information of the source data and the fragmentation rule;
the monitoring module is used for monitoring the performance data of the processing unit and transmitting the performance data to the preprocessing module;
and the preprocessing module is also used for adjusting the source data distributed to the processing unit according to the performance data monitored by the monitoring module.
In one embodiment, the performance data of the processing unit includes: performance parameters and their alarm thresholds;
the adjusting the data allocated to the processing unit according to the monitored performance data of the processing unit includes:
when the performance parameters of the processing unit monitored by the monitoring module reach the alarm threshold value, triggering alarm information and transmitting the alarm information to the preprocessing module;
and the preprocessing module adjusts the transmission interval of the source data distributed to the processing unit according to the alarm information.
In an embodiment, the source packet header further includes: target system information;
the acquiring a target system data template from a target system template set according to the mapping rule and the source data includes:
reading target system information in source data;
and acquiring a target system data template from the target system template set according to the mapping rule and the target system information.
In an embodiment, the processing the source data in the processing unit into the target data according to the target system data template and the mapping rule includes:
unpacking a source packet header and a source packet body of source data, and removing the source packet header;
processing the source data packet body into a target system transaction data packet according to a target system data template according to the mapping rule;
the identification data generated when the processing unit processes the source data is used as a target data packet header;
and packaging the target system transaction data packet as a target data packet body with the target data packet header to generate target data.
In one embodiment, the identification data includes: one or any combination of source system information, target system information, stream information and target system transaction code information.
In an embodiment, the source packet body at least includes: source system transaction code information;
the mapping rule comprises: unique mapping of source system transaction code information to target system transaction code information.
In an embodiment, the mapping rule further includes: a uniquely corresponding target format for the target system transaction code.
In an embodiment, the information of the new target system includes: adding a processing tool for converting the interface type and the entity type of the target system into a target format;
the generating of the data template of the newly added target system according to the information of the newly added target system includes:
acquiring an interface type of a newly added target system, and packaging the interface type in an entity class;
and converting the entity class and the entity class into a processing tool with a target format to form a newly-added target system data template.
The embodiment of the invention also provides computer equipment which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the processor executes the computer program to realize the cross-system data transmission method.
An embodiment of the present invention further provides a computer-readable storage medium, where a computer program for implementing the cross-system data transmission method is stored in the computer-readable storage medium.
To sum up, the cross-system data transmission system and method provided in the embodiments of the present invention adjust data allocation in real time for a processing unit with reduced processing performance by monitoring performance data of the processing unit for cross-system data transmission, so that the processing unit can process data efficiently, and efficiency and performance of data transmission are improved. Furthermore, the embodiment of the invention does not limit the variety increase of the target system any more, and can expand the transaction template set according to different entity transactions and newly added target systems, thereby meeting the data requirements of diversified target systems and enhancing the expandability of data transmission. In the traditional data transmission method, a fixed appointed template is arranged between a source system and a target system, all transaction data are transmitted according to the existing template, and when a new target system and different data acquisition results appear, a data transmission flow needs to be set up again; in the invention, a template set and an extensible framework mode are adopted, and only a new type is required to be newly generated and added into the template set aiming at the data transmission of the new type; in the traditional data transmission method, overall transmission or fixed fragment transmission is mostly adopted, and when the transmission quantity of a certain fragment is increased rapidly, the performance of the fragment is reduced and the regulation is not easy. The invention adopts a distributed architecture, can monitor the performance of a certain fragment in the processing unit in real time through the monitoring module, and can adjust the transmission amount of the fragment by automatically or manually adjusting the transmission interval of the certain fragment, thereby ensuring that each processing unit reaches a high-performance state which is as balanced as possible.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (11)
1. A cross-system data transfer system, comprising: the device comprises a preprocessing module, a data processing module and a data sending module;
the system comprises a pre-processing module, a processing unit and a data processing module, wherein the pre-processing module is used for receiving source data downloaded by a source system, distributing the source data to the processing unit and monitoring performance data of the processing unit; adjusting the source data distributed to the processing units according to the monitored performance data of the processing units;
the data processing module is arranged in the processing unit and used for acquiring a target system data template from the target system template set according to the mapping rule and the source data; processing the source data in the processing unit into target data according to the target system data template and the mapping rule;
the data sending module is used for transmitting the target data to a target system;
the template generation module is used for acquiring the information of the newly added target system when the target system is newly added; generating a data template of the newly added target system according to the information of the newly added target system, and adding the data template to the template set of the target system;
the source data comprises: a source packet header and a source packet body; the source packet header at least comprises: slicing information;
a pre-processing module comprising: the system comprises a preprocessing module and a monitoring module;
the preprocessing module is used for distributing the source data to the processing units according to the fragmentation information of the source data and the fragmentation rule;
the monitoring module is used for monitoring the performance data of the processing unit and transmitting the performance data to the preprocessing module;
and the preprocessing module is also used for adjusting the source data distributed to the processing unit according to the performance data monitored by the monitoring module.
2. The system of claim 1,
performance data of the processing unit, comprising: performance parameters and their alarm thresholds;
the adjusting the data allocated to the processing unit according to the monitored performance data of the processing unit includes:
when the performance parameters of the processing unit monitored by the monitoring module reach the alarm threshold value, triggering alarm information and transmitting the alarm information to the preprocessing module;
and the preprocessing module adjusts the transmission interval of the source data distributed to the processing unit according to the alarm information.
3. The system of claim 1,
the source packet header further includes: target system information;
the acquiring a target system data template from a target system template set according to the mapping rule and the source data includes:
reading target system information in source data;
and acquiring a target system data template from the target system template set according to the mapping rule and the target system information.
4. The system of claim 1,
processing the source data in the processing unit into target data according to the target system data template and the mapping rule, wherein the processing comprises the following steps:
unpacking a source packet header and a source packet body of source data, and removing the source packet header;
processing the source data packet body into a target system transaction data packet according to a target system data template according to the mapping rule;
the identification data generated when the processing unit processes the source data is used as a target data packet header;
and packaging the target system transaction data packet as a target data packet body with the target data packet header to generate target data.
5. The system of claim 4,
the identification data includes: one or any combination of source system information, target system information, stream information and target system transaction code information.
6. The system of claim 5,
the source packet body at least comprises: source system transaction code information;
the mapping rule comprises: unique mapping of source system transaction code information to target system transaction code information.
7. The system of claim 6,
the mapping rule further comprises: a uniquely corresponding target format for the target system transaction code.
8. The system of claim 1,
the information of the newly added target system comprises: adding a processing tool for converting the interface type and the entity type of the target system into a target format;
the generating of the data template of the newly added target system according to the information of the newly added target system includes:
acquiring an interface type of a newly added target system, and packaging the interface type in an entity class;
and converting the entity class and the entity class into a processing tool with a target format to form a newly-added target system data template.
9. A cross-system data transfer method, comprising:
receiving source data downloaded by a source system, distributing the source data to a processing unit, and monitoring performance data of the processing unit; adjusting the source data distributed to the processing unit according to the monitored performance data of the processing unit; the source data includes: a source packet header and a source packet body; the source packet header at least comprises: slicing information;
acquiring a target system data template from the target system template set according to the mapping rule and the source data; processing the source data in the processing unit into target data according to the target system data template and the mapping rule;
transferring the target data to a target system;
when a target system is newly added, acquiring information of the newly added target system; generating a data template of the newly added target system according to the information of the newly added target system, and adding the data template to the template set of the target system;
wherein adjusting the source data allocated to the processing unit according to the monitored performance data of the processing unit comprises:
distributing the source data to the processing unit according to the fragmentation information of the source data and the fragmentation rule;
monitoring performance data of the processing unit;
the source data allocated to the processing unit is adjusted according to the monitored performance data.
10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the cross-system data transfer method of claim 9 when executing the computer program.
11. A computer-readable storage medium storing a computer program for executing a cross-system data transfer method according to claim 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910886758.2A CN110618880B (en) | 2019-09-19 | 2019-09-19 | Cross-system data transmission system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910886758.2A CN110618880B (en) | 2019-09-19 | 2019-09-19 | Cross-system data transmission system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110618880A CN110618880A (en) | 2019-12-27 |
| CN110618880B true CN110618880B (en) | 2022-05-27 |
Family
ID=68923668
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910886758.2A Active CN110618880B (en) | 2019-09-19 | 2019-09-19 | Cross-system data transmission system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110618880B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5909570A (en) * | 1993-12-28 | 1999-06-01 | Webber; David R. R. | Template mapping system for data translation |
| WO2002039649A2 (en) * | 2000-10-20 | 2002-05-16 | Desknet, Inc. | Transforming a source document to a desired target document |
| CN109299126A (en) * | 2018-11-21 | 2019-02-01 | 金蝶软件（中国）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7260777B2 (en) * | 2001-08-17 | 2007-08-21 | Desknet Inc. | Apparatus, method and system for transforming data |
| CN100531055C (en) * | 2006-04-10 | 2009-08-19 | 华为技术有限公司 | Data synchronous system and its method |
| US8086618B2 (en) * | 2008-09-30 | 2011-12-27 | International Business Machines Corporation | Configuration rule translation mapper |
| US9367597B2 (en) * | 2013-11-18 | 2016-06-14 | International Business Machines Corporation | Automatically managing mapping and transform rules when synchronizing systems |
| CN104461516B (en) * | 2014-11-21 | 2018-03-06 | 北京锐安科技有限公司 | A kind of data processing method and device |
| CN104731660B (en) * | 2014-12-31 | 2018-01-12 | 北京天诚盛业科技有限公司 | data distributing method, device and system |
| US20180285151A1 (en) * | 2017-03-31 | 2018-10-04 | Intel Corporation | Dynamic load balancing in network interface cards for optimal system level performance |
| US10764089B2 (en) * | 2017-08-29 | 2020-09-01 | eperi GmbH | Gateway computer system with intermediate data processing according to rules that are specified by templates |
| CN109861797B (en) * | 2017-12-28 | 2022-07-15 | 北京新岸线移动通信技术有限公司 | Data transmission method and system |
| CN109117275A (en) * | 2018-08-31 | 2019-01-01 | 平安科技（深圳）有限公司 | Account checking method, device, computer equipment and storage medium based on data fragmentation |
| CN109144731A (en) * | 2018-08-31 | 2019-01-04 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer equipment and storage medium |
| CN110209496B (en) * | 2019-05-20 | 2022-05-17 | 中国平安财产保险股份有限公司 | Task fragmentation method and device based on data processing and fragmentation server |
-
2019
- 2019-09-19 CN CN201910886758.2A patent/CN110618880B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5909570A (en) * | 1993-12-28 | 1999-06-01 | Webber; David R. R. | Template mapping system for data translation |
| WO2002039649A2 (en) * | 2000-10-20 | 2002-05-16 | Desknet, Inc. | Transforming a source document to a desired target document |
| CN109299126A (en) * | 2018-11-21 | 2019-02-01 | 金蝶软件（中国）有限公司 | Method of data synchronization, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110618880A (en) | 2019-12-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108804545B (en) | Distributed global unique ID generation method and device | |
| CN102523290B (en) | Data processing method, device and system | |
| US20180052789A1 (en) | Direct Memory Access Transmission Control Method and Apparatus | |
| US20200265043A1 (en) | High performance computing system | |
| CN103458270A (en) | Video transcoding system and method | |
| CN105045871A (en) | Data aggregation query method and apparatus | |
| CN104714851A (en) | Method and device for realizing resource distribution | |
| CN110035016A (en) | A kind of data transmission method and device | |
| JP2020129791A5 (en) | ||
| JP2021502006A (en) | Data packet distribution method, transmitting device, receiving device and storage medium | |
| CN105227649A (en) | File transmission method and device | |
| CN111858055A (en) | Task processing method, server and storage medium | |
| CN111641563A (en) | Flow self-adaption method and system based on distributed scene | |
| CN104219298A (en) | Cluster system and data backup method thereof | |
| CN111832883A (en) | Performance assessment system, equipment and medium | |
| CN110618880B (en) | Cross-system data transmission system and method | |
| CN106856459B (en) | Message scheduling method and device | |
| CN111106946A (en) | Method and system for reconstructing edge computing node of networking equipment | |
| EP3506097B1 (en) | Data transmitting program, data transmitting device, and data transmitting method | |
| US10171570B2 (en) | Information processing apparatus | |
| CN109688171B (en) | Cache space scheduling method, device and system | |
| CN112491903B (en) | Account checking method, device and system among multiple systems | |
| CN104699799A (en) | Data transmission method based on cross system | |
| CN106547642B (en) | Data storage backup method and device | |
| CN109510774B (en) | Method for realizing flow control in data synchronization process |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |