CN111538772B - Data exchange processing method and device, electronic equipment and storage medium - Google Patents
Data exchange processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN111538772B CN111538772B CN202010291489.8A CN202010291489A CN111538772B CN 111538772 B CN111538772 B CN 111538772B CN 202010291489 A CN202010291489 A CN 202010291489A CN 111538772 B CN111538772 B CN 111538772B
- Authority
- CN
- China
- Prior art keywords
- data
- node
- source
- data exchange
- plug
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The embodiment of the invention discloses a data exchange processing method, a device, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring a source node and a destination node according to the received data exchange request, and determining a data exchange link according to the source node and the destination node; acquiring source data in a source node through a data exchange plug-in, and integrating the source data to obtain processed data; and determining a transfer node according to the data exchange link, and sending the processed data to the transfer node so that the transfer node directly or indirectly forwards the processed data to a destination node. The source data are integrated and processed through the data exchange plug-in unit and then sent to the transfer node, and different source data are subjected to data conversion in a plug-in mode, so that a scene of mutually and freely exchanging various types of data can be supported, data exchange in heterogeneous networks and cross-network environments can be supported, and meanwhile, when mass data are faced, the practicability and the compression resistance of the data can be greatly improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data exchange processing method and apparatus, an electronic device, and a storage medium.
Background
With the popularity of computer technology and network technology, there is an increasing demand for data exchange. Data exchange refers to the exchange of data from a source node to a destination node. The source node is the node where the data source is located, and the data source may be a database (relational: mysql, oralce, sqlserver, general, personal Jin Cang, etc., non-relational: hbase, mongodb, etc.), a file (csv file, txt file, etc.), and an entity that may be stored in some manner.
The data exchange method in the prior art is not comprehensive, single, and can not simultaneously meet the free exchange among different types of data; meanwhile, for the scenes of heterogeneous networks and cross-network environments, no good solution exists. In an actual production environment, massive data are often required to face, the requirement on the implementation of the data is high, and the compression resistance is high.
Disclosure of Invention
Because the existing method has the problems, the embodiment of the invention provides a data exchange processing method, a device, electronic equipment and a storage medium.
In a first aspect, an embodiment of the present invention provides a data exchange processing method, including:
if a data exchange request is received, acquiring a source node and a destination node according to the data exchange request, and determining a data exchange link according to the source node and the destination node;
acquiring the source data in the source node through a data exchange plug-in, and integrating the source data to obtain processed data;
determining a transfer node according to the data exchange link, and sending the processed data to the transfer node, so that the transfer node directly or indirectly forwards the processed data to the destination node, and writes the processed data into the destination node;
the source node is a node where the source data is located, and the destination node is a node where the destination data is located.
Optionally, if a data exchange request is received, acquiring a source node and a destination node according to the data exchange request, and determining a data exchange link according to the source node and the destination node, including:
and if the data exchange request is received, acquiring a source node and a destination node according to the data exchange request, and determining the shortest data exchange link between the source node and the destination node according to a shortest path algorithm.
Optionally, the acquiring the source data in the source node through the data exchange plug-in unit, and after integrating the source data, obtaining processed data specifically includes:
and acquiring the source data in the source node through a data exchange plug-in, and respectively extracting, converting and loading the source data to obtain integrated processed data.
Optionally, the data exchange plug-in includes a read sub-plug-in, a transmit sub-plug-in, and a write sub-plug-in.
Optionally, the data exchange processing method further includes:
and determining a corresponding data exchange plug-in according to the type of the source data.
Optionally, the acquiring, by a data switch, the source data in the source node specifically includes:
acquiring the source data in the source node through the reading sub-plug-in;
the determining a transfer node according to the data exchange link, and sending the processed data to the transfer node, so that the transfer node directly or indirectly forwards the processed data to the destination node, and writes the processed data into the destination node, specifically including:
and determining a transfer node according to the data exchange link, sending the processed data to the transfer node through the transmission sub-plug-in, enabling the transfer node to directly or indirectly forward the processed data to the destination node, and writing the processed data into the destination node through the writing sub-plug-in.
Optionally, the data exchange processing method further includes:
acquiring monitoring data, and if abnormal data exist in the monitoring data, performing abnormal processing on the abnormal data according to a preset abnormal data processing rule.
In a second aspect, an embodiment of the present invention further provides a data exchange processing apparatus, including:
the link determining module is used for acquiring a source node and a destination node according to the data exchange request if the data exchange request is received, and determining a data exchange link according to the source node and the destination node;
the data integration module is used for acquiring the source data in the source node through a data exchange plug-in, and integrating the source data to obtain processed data;
the data forwarding module is used for determining a transit node according to the data exchange link and sending the processed data to the transit node so that the transit node forwards the processed data directly or indirectly to the destination node and writes the processed data into the destination node;
the source node is a node where the source data is located, and the destination node is a node where the destination data is located.
Optionally, the link determining module is specifically configured to obtain the source node and the destination node according to the data exchange request if the data exchange request is received, and determine a shortest data exchange link between the source node and the destination node according to a shortest path algorithm.
Optionally, the data integration module is specifically configured to obtain the source data in the source node through a data exchange plug-in, and perform extraction, conversion and loading processing on the source data respectively to obtain integrated processed data.
Optionally, the data exchange plug-in includes a read sub-plug-in, a transmit sub-plug-in, and a write sub-plug-in.
Optionally, the data exchange processing device further includes:
and the plug-in determining module is used for determining a corresponding data exchange plug-in according to the type of the source data.
Optionally, the data integration module is specifically configured to obtain, through the reading sub-plug-in, the source data in the source node;
the data forwarding module is specifically configured to determine a transit node according to the data exchange link, and send the processed data to the transit node through the transport sub-plug-in, so that the transit node forwards the processed data directly or indirectly to the destination node, and writes the processed data into the destination node through the write sub-plug-in.
Optionally, the data exchange processing device further includes:
the abnormal processing module is used for acquiring the monitoring data, and if the monitoring data are judged to have abnormal data, the abnormal data are subjected to abnormal processing according to a preset abnormal data processing rule.
In a third aspect, an embodiment of the present invention further provides an electronic device, including:
at least one processor; and
at least one memory communicatively coupled to the processor, wherein:
the memory stores program instructions executable by the processor, which are called by the processor to perform the method described above.
In a fourth aspect, embodiments of the present invention also propose a non-transitory computer-readable storage medium storing a computer program, which causes the computer to carry out the above-mentioned method.
According to the technical scheme, the source data are integrated through the data exchange plug-in unit and then sent to the transfer node, so that the destination node is finally achieved, different source data are subjected to data conversion through the plug-in unit, a scene that multiple types of data are freely exchanged with each other can be supported, data exchange in heterogeneous networks and cross-network environments can be supported, and meanwhile, when the mass data are faced, the practicability and the compression resistance of the data can be greatly improved.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings that are necessary for the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention and that other drawings can be obtained from these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a data exchange processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating data exchange through a data exchange plug-in according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a data exchange processing device according to an embodiment of the present invention;
fig. 4 is a logic block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following describes the embodiments of the present invention further with reference to the accompanying drawings. The following examples are only for more clearly illustrating the technical aspects of the present invention, and are not intended to limit the scope of the present invention.
Fig. 1 shows a flow chart of a data exchange processing method provided in this embodiment, including:
s101, if a data exchange request is received, acquiring a source node and a destination node according to the data exchange request, and determining a data exchange link according to the source node and the destination node.
The data exchange request is a request which is received by a current source node and needs to be subjected to data exchange.
The source node is the node where the source data is located.
The destination node is the node where the destination data is located.
The data exchange link is a link composed of data nodes through which data exchange is performed.
S102, acquiring the source data in the source node through a data exchange plug-in, and integrating the source data to obtain processed data.
The data exchange plug-in is a plug-in which is additionally arranged on each data node and is used for realizing data exchange and supports hot plug.
The processed data are data obtained after the source data are integrated.
And S103, determining a transfer node according to the data exchange link, and sending the processed data to the transfer node so that the transfer node directly or indirectly forwards the processed data to the destination node and writes the processed data into the destination node.
The transfer node is other intermediate nodes except the source node and the destination node in the data exchange link.
Specifically, if one transit node exists in the data exchange link, the transit node directly forwards the processed data to a destination node; if there are multiple transfer nodes in the data exchange link, the current transfer node forwards the processed data to the next transfer node, and each transfer node continuously forwards the processed data until the processed data is sent to the destination node.
According to the embodiment, after the source data are integrated through the data exchange plug-in, the source data are sent to the transfer node, the destination node is finally reached, different source data are subjected to data conversion in a plug-in mode, a scene that multiple types of data are mutually and freely exchanged can be supported, data exchange in heterogeneous networks and cross-network environments can also be supported, and meanwhile when massive data are faced, the practicability and the compression resistance of the data can be greatly improved.
Further, on the basis of the above method embodiment, S101 specifically includes:
and if the data exchange request is received, acquiring a source node and a destination node according to the data exchange request, and determining the shortest data exchange link between the source node and the destination node according to a shortest path algorithm.
The shortest path algorithm is an algorithm for calculating the minimum transfer node between the source node and the destination node.
In the embodiment, the shortest data exchange link between the source node and the destination node is determined through the shortest path algorithm, so that the data exchange speed is improved.
Further, on the basis of the above method embodiment, S102 specifically includes:
and acquiring the source data in the source node through a data exchange plug-in, and respectively extracting, converting and loading the source data to obtain integrated processed data.
Specifically, after the source data in the source node is acquired through a data switch, ETL (Extract-Transform-Load) processing may be performed on the data.
According to the embodiment, the data is loaded into the data warehouse after being extracted, washed and converted, so that scattered, disordered and non-uniform data can be integrated together, and an analysis basis is provided for subsequent decisions.
Further, on the basis of the above method embodiment, the data exchange processing method further includes:
and determining a corresponding data exchange plug-in according to the type of the source data.
Wherein the source data types include various data types, such as character type data, text type data or other custom type data.
Specifically, since the data exchange in the prior art is single, the scenario that multiple types of data are freely exchanged with each other cannot be satisfied, in this embodiment, data conversion is performed on different source data by means of plug-ins, and a corresponding data exchange plug-in is determined for each type of source data, and through the data exchange plug-in, real-time data exchange between a source node and a destination node is realized.
Further, on the basis of the above method embodiment, the data exchange processing method further includes:
acquiring monitoring data, and if abnormal data exist in the monitoring data, performing abnormal processing on the abnormal data according to a preset abnormal data processing rule.
The monitoring data are all data monitored in the execution process of the embodiment.
The abnormal data is monitoring data with abnormal conditions, such as data conversion failure.
The abnormal data processing rule is a rule which is preset according to requirements and is used for processing abnormal data, for example, the abnormal data is reported to a target node for analysis and statistics by the target node.
Specifically, the data exchange processing method provided in this embodiment may include the following specific steps:
a1, a user sets a source node and a destination node, and uniformly sets related processing operation of data, flow task information and the like;
and A2, after the data exchange request is issued, the system intelligently searches the shortest and best data exchange link according to the shortest path algorithm. Firstly, a Mysqlread plug-in reads data from a data source, and carries out ETL processing on the data according to the setting of a user;
a3, forwarding the data according to the sequence of the data exchange links, and finally reaching a destination node;
and A4, reporting the monitoring data by the system, recording the abnormal data, and processing the abnormal data according to the user setting.
The embodiment supports the scene of mutually freely exchanging various types of data, supports the data exchange in heterogeneous networks and cross-network environments, and can timely capture and process corresponding abnormal data when abnormal data occurs.
Further, on the basis of the foregoing method embodiment, the acquiring, in S102, the source data in the source node through a data switch plug-in specifically includes:
acquiring the source data in the source node through the reading sub-plug-in;
s103 specifically comprises the following steps:
and determining a transfer node according to the data exchange link, sending the processed data to the transfer node through the transmission sub-plug-in, enabling the transfer node to directly or indirectly forward the processed data to the destination node, and writing the processed data into the destination node through the writing sub-plug-in.
The data exchange plug-in comprises a reading sub-plug-in, a transmitting sub-plug-in and a writing sub-plug-in.
Specifically, the data exchange plug-in provided in this embodiment is composed of three parts, namely a read sub-plug-in reader, a transmit sub-plug-in channel and a write sub-plug-in writer, and there are corresponding readers and writers for different data types. The construction is performed based on a producer consumer model, wherein a reader is responsible for reading data from cluster source data, a channel is an intermediate layer and responsible for landing of the data, and a writer is responsible for writing the data into a target data source.
Because the existing data exchange method faces mass data, has no good compression resistance, and has the problems of low timeliness, low data source expansibility, no support for plug deployment, no support for intelligent processing, easy data loss and the like, an interaction schematic diagram of data exchange through a data exchange plug-in provided by the embodiment is shown in fig. 2, and data is exchanged from a source node (src node) to a destination node (dst node) through a relay node (replay node). Fig. 2 is a cross-network switching scenario, i.e. a network between the cross-level nodes is not connected, and only the upper and lower level nodes are connected to each other.
The Src node is a source node, namely a data node to be exchanged; the Replay node is a transfer node and is responsible for bridging different nodes of the network and plays a role of data transfer; dst node is the destination node, i.e., the final destination node of the destination data.
Reader: different plug-ins are set for different data sources, such as mysql reader, oracle reader, hbase reader, etc. Is responsible for acquiring data from interactions with the source data source.
Channel: and the data is responsible for data persistence and pushing, and consists of a channel reader, a channel writer and a data channel. The channel reader is responsible for reading data from channel data; channel data is responsible for storing data and persisting the data; the channel writer is responsible for writing data into the channel data.
Writer: different plug-ins are set for different data sources, such as mysql writer, oracle writer, hbase writer, etc. Is responsible for exchanging data from and to the destination data source.
Different plugins can be customized for different data sources, and when a switching flow task is started, a mysql reader plugin in a source node can read source data from a mysql database, ETL processing is carried out on the data, a channel writer writes the data into a channel, and then the channel reader can read the data from the channel and send the data to a transit node channel through the writer. On the transfer node, the channel reader reads the channel data, and the writer writes the data into the channel of the destination node. And the channel reader takes the data out of the channel at the destination node, and the hbase writer writes the data into the hbase until the whole data exchange flow is finished.
According to the embodiment, after the source data are integrated through the data exchange plug-in, the source data are sent to the transfer node, the destination node is finally reached, different source data are subjected to data conversion in a plug-in mode, a scene that multiple types of data are mutually and freely exchanged can be supported, data exchange in heterogeneous networks and cross-network environments can also be supported, and meanwhile when massive data are faced, the practicability and the compression resistance of the data can be greatly improved.
Fig. 3 shows a schematic structural diagram of a data exchange processing apparatus according to the present embodiment, where the apparatus includes: a link determination module 301, a data integration module 302 and a data forwarding module 303, wherein:
the link determining module 301 is configured to obtain a source node and a destination node according to a data exchange request if the data exchange request is received, and determine a data exchange link according to the source node and the destination node;
the data integration module 302 is configured to obtain the source data in the source node through a data switch, and perform integration processing on the source data to obtain processed data;
the data forwarding module 303 is configured to determine a transit node according to the data exchange link, and send the processed data to the transit node, so that the transit node forwards the processed data directly or indirectly to the destination node, and writes the processed data into the destination node;
the source node is a node where the source data is located, and the destination node is a node where the destination data is located.
Specifically, if the link determining module 301 receives a data exchange request, it obtains a source node and a destination node according to the data exchange request, and determines a data exchange link according to the source node and the destination node; the data integration module 302 obtains the source data in the source node through a data exchange plug-in, and obtains processed data after integrating the source data; the data forwarding module 303 determines a forwarding node according to the data exchange link, and sends the processed data to the forwarding node, so that the forwarding node forwards the processed data directly or indirectly to the destination node, and writes the processed data into the destination node.
According to the embodiment, after the source data are integrated through the data exchange plug-in, the source data are sent to the transfer node, the destination node is finally reached, different source data are subjected to data conversion in a plug-in mode, a scene that multiple types of data are mutually and freely exchanged can be supported, data exchange in heterogeneous networks and cross-network environments can also be supported, and meanwhile when massive data are faced, the practicability and the compression resistance of the data can be greatly improved.
Further, on the basis of the above device embodiment, the link determining module 301 is specifically configured to obtain, if a data exchange request is received, a source node and a destination node according to the data exchange request, and determine, according to a shortest path algorithm, a shortest data exchange link between the source node and the destination node.
Further, on the basis of the above device embodiment, the data integration module 302 is specifically configured to obtain the source data in the source node through a data switch, and perform processing of extracting, converting, and loading on the source data, so as to obtain integrated processed data.
Further, on the basis of the above device embodiment, the data exchange plug-in includes a read sub-plug-in, a transmit sub-plug-in, and a write sub-plug-in.
Further, on the basis of the above device embodiment, the data exchange processing device further includes:
and the plug-in determining module is used for determining a corresponding data exchange plug-in according to the type of the source data.
Further, on the basis of the above device embodiment, the data integration module 302 is specifically configured to obtain, through the reading sub-plug-in, the source data in the source node;
the data forwarding module is specifically configured to determine a transit node according to the data exchange link, and send the processed data to the transit node through the transport sub-plug-in, so that the transit node forwards the processed data directly or indirectly to the destination node, and writes the processed data into the destination node through the write sub-plug-in.
Further, on the basis of the above device embodiment, the data exchange processing device further includes:
the abnormal processing module is used for acquiring the monitoring data, and if the monitoring data are judged to have abnormal data, the abnormal data are subjected to abnormal processing according to a preset abnormal data processing rule.
The data exchange processing device in this embodiment may be used to execute the method embodiment, and the principle and technical effects are similar, and are not described herein.
Referring to fig. 4, the electronic device includes: a processor (processor) 401, a memory (memory) 402, and a bus 403;
wherein, the liquid crystal display device comprises a liquid crystal display device,
the processor 401 and the memory 402 complete communication with each other through the bus 403;
the processor 401 is configured to call the program instructions in the memory 402 to perform the methods provided in the above method embodiments.
The present embodiments disclose a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, are capable of performing the methods provided by the method embodiments described above.
The present embodiment provides a non-transitory computer-readable storage medium storing computer instructions that cause a computer to perform the methods provided by the above-described method embodiments.
The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method described in the respective embodiments or some parts of the embodiments.
It should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (9)
1. A data exchange processing method, characterized by comprising:
if a data exchange request is received, acquiring a source node and a destination node according to the data exchange request, and determining a data exchange link according to the source node and the destination node;
determining a corresponding data exchange plug-in according to the type of the source data; acquiring the source data in the source node through a data exchange plug-in, and integrating the source data to obtain processed data;
determining a transfer node according to the data exchange link, and sending the processed data to the transfer node, so that the transfer node directly or indirectly forwards the processed data to the destination node, and writes the processed data into the destination node;
the source node is a node where the source data is located, and the destination node is a node where the destination data is located.
2. The method according to claim 1, wherein if a data exchange request is received, acquiring a source node and a destination node according to the data exchange request, and determining a data exchange link according to the source node and the destination node, specifically comprising:
and if the data exchange request is received, acquiring a source node and a destination node according to the data exchange request, and determining the shortest data exchange link between the source node and the destination node according to a shortest path algorithm.
3. The method for exchanging data according to claim 1, wherein the obtaining the source data in the source node through the data exchange plug-in unit, and performing an integration process on the source data to obtain processed data, specifically includes:
and acquiring the source data in the source node through a data exchange plug-in, and respectively extracting, converting and loading the source data to obtain integrated processed data.
4. A data exchange processing method according to claim 3, wherein the data exchange plug-in includes a read sub-plug-in, a transmit sub-plug-in and a write sub-plug-in.
5. The method for processing data according to claim 4, wherein the obtaining the source data in the source node through a data switch module specifically includes:
acquiring the source data in the source node through the reading sub-plug-in;
the determining a transfer node according to the data exchange link, and sending the processed data to the transfer node, so that the transfer node directly or indirectly forwards the processed data to the destination node, and writes the processed data into the destination node, specifically including:
and determining a transfer node according to the data exchange link, sending the processed data to the transfer node through the transmission sub-plug-in, enabling the transfer node to directly or indirectly forward the processed data to the destination node, and writing the processed data into the destination node through the writing sub-plug-in.
6. The exchange processing method of data according to any one of claims 1 to 4, wherein the exchange processing method of data further comprises:
acquiring monitoring data, and if abnormal data exist in the monitoring data, performing abnormal processing on the abnormal data according to a preset abnormal data processing rule.
7. A data exchange processing apparatus, comprising:
the link determining module is used for acquiring a source node and a destination node according to the data exchange request if the data exchange request is received, and determining a data exchange link according to the source node and the destination node;
the data integration module is used for determining a corresponding data exchange plug-in according to the type of the source data; acquiring the source data in the source node through a data exchange plug-in, and integrating the source data to obtain processed data;
the data forwarding module is used for determining a transit node according to the data exchange link and sending the processed data to the transit node so that the transit node forwards the processed data directly or indirectly to the destination node and writes the processed data into the destination node;
the source node is a node where the source data is located, and the destination node is a node where the destination data is located.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the data exchange processing method according to any one of claims 1 to 6 when executing the program.
9. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, implements the data exchange processing method according to any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010291489.8A CN111538772B (en) | 2020-04-14 | 2020-04-14 | Data exchange processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010291489.8A CN111538772B (en) | 2020-04-14 | 2020-04-14 | Data exchange processing method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111538772A CN111538772A (en) | 2020-08-14 |
| CN111538772B true CN111538772B (en) | 2023-07-04 |
Family
ID=71978693
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010291489.8A Active CN111538772B (en) | 2020-04-14 | 2020-04-14 | Data exchange processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111538772B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112165529A (en) * | 2020-09-30 | 2021-01-01 | 南威软件股份有限公司 | Method, device, equipment and medium for low-cost cross-network data exchange |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108572994A (en) * | 2017-03-14 | 2018-09-25 | 中国移动通信集团公司 | A kind of processing method and server of Data Migration |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102170440B (en) * | 2011-03-24 | 2013-12-04 | 北京大学 | Method suitable for safely migrating data between storage clouds |
| CN103023729A (en) * | 2011-09-22 | 2013-04-03 | 中兴通讯股份有限公司 | Relay node selection method and device |
| CN102693297B (en) * | 2012-05-16 | 2015-03-11 | 华为技术有限公司 | Data processing method, node and ETL (extract transform and load) system |
| CN104699723B (en) * | 2013-12-10 | 2018-10-19 | 北京神州泰岳软件股份有限公司 | Data synchronous system and method between data exchange adapter, heterogeneous system |
| US9613190B2 (en) * | 2014-04-23 | 2017-04-04 | Intralinks, Inc. | Systems and methods of secure data exchange |
| CN104111834A (en) * | 2014-07-07 | 2014-10-22 | 北京京东尚科信息技术有限公司 | Method and device for exchanging data between heterogeneous databases |
| CN106844399B (en) * | 2015-12-07 | 2022-08-09 | 中兴通讯股份有限公司 | Distributed database system and self-adaptive method thereof |
| CN106250571A (en) * | 2016-10-11 | 2016-12-21 | 北京集奥聚合科技有限公司 | The method and system that a kind of ETL data process |
| CN108304473B (en) * | 2017-12-28 | 2020-09-04 | 石化盈科信息技术有限责任公司 | Data transmission method and system between data sources |
| CN108388615B (en) * | 2018-02-09 | 2019-07-23 | 杭州数梦工场科技有限公司 | A kind of method for interchanging data, system and electronic equipment |
| CN109327387A (en) * | 2018-11-29 | 2019-02-12 | 甘肃万维信息技术有限责任公司 | Data share exchange system and method based on dynamic routing |
| CN110909079B (en) * | 2019-11-20 | 2020-09-01 | 南方电网数字电网研究院有限公司 | Data exchange synchronization method, system, device, server and storage medium |
-
2020
- 2020-04-14 CN CN202010291489.8A patent/CN111538772B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108572994A (en) * | 2017-03-14 | 2018-09-25 | 中国移动通信集团公司 | A kind of processing method and server of Data Migration |
Non-Patent Citations (1)
| Title |
|---|
| 一种适用于非对称链路的AdHoc网络路由协议;陈晋伦,吕国英,周正;电路与系统学报(06);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111538772A (en) | 2020-08-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9717011B2 (en) | Event management in telecommunications networks | |
| KR101871383B1 (en) | Method and system for using a recursive event listener on a node in hierarchical data structure | |
| CN108696297B (en) | Equipment state detection method and device in power line communication network | |
| WO2016017208A1 (en) | Monitoring system, monitoring device, and inspection device | |
| CN109271363B (en) | File storage method and device | |
| CN109450936A (en) | A kind of adaptation method and device of the hetero-com-munication agreement based on Kafka | |
| JP2013243670A (en) | Packet processing method, device and system | |
| CN111966289A (en) | Partition optimization method and system based on Kafka cluster | |
| CN108322318B (en) | Alarm analysis method and equipment | |
| CN108132868A (en) | A kind of data monitoring method, device, computing device and storage medium | |
| CN111538772B (en) | Data exchange processing method and device, electronic equipment and storage medium | |
| WO2023061329A1 (en) | Communication method applied to computer cluster, and communication apparatus | |
| CN116662022A (en) | Distributed message processing method, system, device, communication equipment and storage medium | |
| JP6364727B2 (en) | Information processing system, distributed processing method, and program | |
| CN114706829A (en) | Big data processing method and system applied to meta universe | |
| CN107493308B (en) | Method and device for sending message and distributed equipment cluster system | |
| JP6432377B2 (en) | Message log removing apparatus, message log removing method, and message log removing program | |
| CN109143878B (en) | Remote signaling message and SOE message processing method for multi-channel acquisition channel | |
| JP2019009610A (en) | Edge device, data processing system, data transmission method and program | |
| CN113434604A (en) | Time sequence database synchronization method, system, equipment and storage medium | |
| CN112468317A (en) | Cluster topology updating method, system, equipment and computer storage medium | |
| CN112488836B (en) | Transaction transmitting method, device, electronic equipment and readable storage medium | |
| WO2022105374A1 (en) | Information processing method, method for generating and training module, electronic device, and medium | |
| CN117395747B (en) | Unmanned aerial vehicle remote control networking method and system | |
| JP5962196B2 (en) | Monitoring server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |