CN113760923A - Data heterogeneous method, device, system and storage medium - Google Patents
Data heterogeneous method, device, system and storage medium Download PDFInfo
- Publication number
- CN113760923A CN113760923A CN202011165198.0A CN202011165198A CN113760923A CN 113760923 A CN113760923 A CN 113760923A CN 202011165198 A CN202011165198 A CN 202011165198A CN 113760923 A CN113760923 A CN 113760923A
- Authority
- CN
- China
- Prior art keywords
- data
- heterogeneous
- monitoring point
- target
- isomerism
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- Computer Security & Cryptography (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention discloses a data heterogeneous method, a device, a system and a storage medium. The method is applied to a target service system, a target data source and a distributed task are arranged in the target service system, and the method comprises the following steps: when monitoring a monitoring point generation event from a distributed task, acquiring a monitoring point corresponding to the monitoring point generation event; acquiring heterogeneous data corresponding to the monitoring point in the target data source, and verifying the heterogeneous data based on the monitoring point; and if the data after the isomerism is determined to be inconsistent with the data before the isomerism corresponding to the monitoring point according to the checking result, updating the data after the isomerism according to the data before the isomerism. According to the technical scheme of the embodiment of the invention, the heterogeneous service data in the target data source is verified through the monitoring point generated in the data heterogeneous process, and the data updating operation is executed when the service data is determined to be inconsistent in the two data sources according to the verification result, so that the consistency of the heterogeneous service data in the two data sources is ensured.
Description
Technical Field
The embodiment of the invention relates to the technical field of computers, in particular to a data heterogeneous method, a data heterogeneous device, a data heterogeneous system and a storage medium.
Background
Data isomerism is applied to many application scenarios, for example, data isomerism exists from order ID sub-database sub-tables to merchant ID sub-database sub-tables, multiple tables are combined into one table, and the like.
In practical application, when some service data is heterogeneous from an original data source to a target data source, and when the original data source is mysql, an interaction protocol of a slave server (slave) of mysql can be simulated based on a component subscribing to a binlog log of an open source, and the slave server (slave) can pretend to be slave, so that the binlog can be read from a master server (master) of the mysql, and a read result is stored in the target data source.
In the process of implementing the invention, the inventor finds that the following technical problems exist in the prior art: the target service system where the target data source is located may also have some service logics, and the target service system may change service data heterogeneous from the original data source to the target data source, where the change causes the heterogeneous service data to be inconsistent in the original data source and the target data source.
Disclosure of Invention
The embodiment of the invention provides a data heterogeneous method, a device, a system and a storage medium, which solve the problem that heterogeneous service data may be inconsistent in two data sources.
In a first aspect, an embodiment of the present invention provides a data heterogeneous method, where the method is applicable to a target service system, and the target service system is provided with a target data source and a distributed task, and the method includes:
when monitoring a monitoring point generation event from a distributed task, acquiring a monitoring point corresponding to the monitoring point generation event, wherein the monitoring point is a record newly added in the distributed task when data in an original data source is heterogeneous to a target data source before the data is heterogeneous; acquiring heterogeneous data corresponding to the monitoring point in the target data source, and verifying the heterogeneous data based on the monitoring point; and if the data after the isomerism is determined to be inconsistent with the data before the isomerism corresponding to the monitoring point according to the checking result, updating the data after the isomerism according to the data before the isomerism.
Optionally, the monitoring point is configured with a preset time window, and checking the heterogeneous data based on the monitoring point may include: and when the monitoring point generates a preset time window after the monitoring time point of the event, checking the heterogeneous data based on the monitoring point.
Optionally, verifying the heterogeneous data based on the monitoring point may include: if the newly added state information in the monitoring point and the heterogeneous state information in the heterogeneous data are both preset state information, respectively acquiring a newly added time point when the monitoring point is newly added to the distributed task and a heterogeneous time point when the heterogeneous data is heterogeneous to the target data source; and acquiring a current time point, and comparing with the newly added time point, if the heterogeneous time point is closer to the current time point, checking the heterogeneous data according to whether the time difference value between the heterogeneous time point and the newly added time point is greater than a preset time threshold value.
Optionally, verifying the heterogeneous data based on the monitoring point may include: acquiring the preset appearance sequence of each state information; and if the newly added state information in the monitoring point and the heterogeneous state information in the heterogeneous data belong to different original service systems, checking the heterogeneous data according to the relative positions of the newly added state information and the heterogeneous state information in the sequence of occurrence in the first and the last steps.
Optionally, the data heterogeneous method may further include:
when a data updating event derived from a theme in the MQ is monitored, consuming the MQ according to the data updating event; writing the heterogeneous pre-data consumed from the MQ into a target data source, and generating a monitoring point in the distributed task according to a consumption result;
correspondingly, updating the post-isomerism data according to the pre-isomerism data may include: consuming the MQ again to obtain data before isomerism; and updating the data after the isomerism according to the obtained data before the isomerism.
On this basis, optionally, writing the pre-heterogeneous data consumed from the MQ into the target data source may include: and doubly writing the heterogeneous pre-data consumed from the MQ into a first data table and a target data source, wherein the first data table stores the full information of each item.
Optionally, the data heterogeneous method may further include: when a data change event from a target data source is monitored, determining heterogeneous state information in heterogeneous data corresponding to the data change event; and if the business process node corresponding to the data after the isomerization is determined not to be the business end node according to the isomerization state information, isomerizing the data after the isomerization into a second data table.
Optionally, the data heterogeneous method may further include: when a data sampling event is monitored, determining a data sampling condition according to the data sampling event; respectively acquiring a first sampling result in a target data source and a second sampling result in an existing data source according to a data sampling condition; and taking the first sampling result and the second sampling result as a group of verification sampling results, and verifying the correctness of each service data in the target data source according to the plurality of groups of verification sampling results.
Optionally, the original business system where the original data source is located includes a warehousing system and/or a distribution system, and the target business system includes a reporting system.
In a second aspect, an embodiment of the present invention further provides a data heterogeneous device, where the device is configured in a target service system, and the target service system is provided with a target data source and a distributed task, and the device includes:
the monitoring point acquisition module is used for acquiring a monitoring point corresponding to a monitoring point generation event when monitoring the monitoring point generation event from the distributed task, wherein the monitoring point is a record newly added in the distributed task when heterogeneous pre-data in an original data source is heterogeneous to a target data source;
the data verification module is used for acquiring heterogeneous data corresponding to the monitoring point in the target data source and verifying the heterogeneous data based on the monitoring point;
and the data updating module is used for updating the data after the isomerism according to the data before the isomerism if the data after the isomerism is determined to be inconsistent with the data before the isomerism corresponding to the monitoring point according to the verification result.
In a third aspect, an embodiment of the present invention further provides a target service system, which may include:
one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the data heterogeneous method provided by any embodiment of the invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the data heterogeneous method provided in any embodiment of the present invention.
According to the technical scheme of the embodiment of the invention, when a target service system monitors a monitoring point generation event from a distributed task, a monitoring point corresponding to the monitoring point generation event can be obtained, namely a record newly added in the distributed task when service data are heterogeneous from an original data source to a target data source is obtained; and then, checking heterogeneous data corresponding to the monitoring point in a target data source based on the monitoring point, and if the heterogeneous data is determined to be inconsistent with the pre-heterogeneous data in the original data source corresponding to the monitoring point according to a checking result, updating the heterogeneous data according to the pre-heterogeneous data. According to the technical scheme, the heterogeneous service data in the target data source is verified through the corresponding monitoring points generated in the data heterogeneous process, and the data updating operation is executed when the service data is determined to be inconsistent in the two data sources according to the verification result, so that the consistency of the heterogeneous service data in the two data sources is guaranteed.
Drawings
Fig. 1 is a flowchart of a data heterogeneous method according to a first embodiment of the present invention;
FIG. 2 is a flowchart of a data heterogeneous method according to a second embodiment of the present invention;
fig. 3 is a flowchart of a data heterogeneous method according to a third embodiment of the present invention;
fig. 4a is a schematic diagram of a first architecture in a data heterogeneous method according to a third embodiment of the present invention;
fig. 4b is a schematic diagram of a second architecture in a data heterogeneous method according to a third embodiment of the present invention;
fig. 4c is a schematic diagram of a third architecture in a data heterogeneous method according to a third embodiment of the present invention;
fig. 5 is a block diagram of a data heterogeneous apparatus according to a fourth embodiment of the present invention;
fig. 6 is a schematic structural diagram of a target service system in the fifth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Before the embodiment of the present invention is described, an application scenario of the embodiment of the present invention is exemplarily described: the data heterogeneous method can be applied to a target service system, a target data source can be arranged in the target service system, certain service data can be heterogeneously from an original data source to the target data source, the service data can be called as pre-heterogeneous data in the original data source, namely the pre-heterogeneous data is service data before heterogeneous; the service data may be referred to as heterogeneous data in the target data source, that is, the heterogeneous data is heterogeneous service data. The original data source and the target data source may be any data source, such as mysql, DB, Elastic Search (ES), etc., and some traffic data may be heterogeneous from the original data source to the target data source in a number of ways, such as binlog consumption, MQ, etc.
Considering the e-commerce platform scenario that may be involved in the embodiments of the present invention, the original business system where the original data source is located may be a warehousing system, a distribution system, and the like, the target business system may be a reporting system, and the like, and the business data in the warehousing system and/or the distribution system may be heterogeneous to the reporting system, because the warehousing system usually stores the business data related to the production logic, such as the change of the ex-warehouse time and the inventory information when the order is ex-warehouse, and the distribution system usually stores the business data related to the distribution logic, such as the distribution time point when the order is distributed, and these business data may be stored based on a sub-warehouse and sub-table manner, which means that the business data in a single warehousing system or distribution system cannot perfect the data reporting information in the reporting system, and it needs to merge the business data in the warehousing system and the distribution system to obtain complete data reporting information, the data report information can be represented by a data table and an index table, i.e. various service data belonging to the same order number are combined into a data chain, so that an operator can analyze the operation data according to the data chain, and a customer can inquire the order state according to the data chain.
The target service system can also be provided with a distributed task, the distributed task can be regarded as a storage unit, a log unit and the like, and when the target service system monitors that the service data in the original data source is changed in a certain mode, the target service system can record the service data in the distributed task besides isomerizing the service data to the target data source. Optionally, each line in the distributed task may record only one service data, that is, one service data is recorded in a new line for the next service data, and operations such as merging of the service data are not performed. Continuing with the e-commerce platform scenario described above as an example, each record in the distributed task may be all processes that have occurred for the order corresponding to each order number, such as the order being out of stock, delivered, committed, etc. at a certain time. Optionally, each row in the distributed task may record only one state change of a certain order, for example, a state of a certain order is changed from a state a to a state B at a certain time, and different state changes of the same order may be recorded in different rows.
Example one
Fig. 1 is a flowchart of a data heterogeneous method according to a first embodiment of the present invention. The embodiment can be applied to the condition of checking the consistency of the data after the isomerism and the data before the isomerism. The method can be executed by a data heterogeneous device provided by the embodiment of the invention, the device can be realized by software and/or hardware, and the device can be integrated in a target business system.
Referring to fig. 1, the method of the embodiment of the present invention specifically includes the following steps:
s110, when monitoring a monitoring point generation event from the distributed task, acquiring a monitoring point corresponding to the monitoring point generation event, wherein the monitoring point is a record newly added in the distributed task when data before isomerization in an original data source is isomerized to a target data source.
As described above, when a service data is heterogeneous from an original data source to a target data source, it may also be recorded in a distributed task, where the service data may be referred to as pre-heterogeneous data in the original data source and may be referred to as post-heterogeneous data in the target data source. The process of adding a record in the distributed task may be understood as a process of generating a monitoring point in the distributed task, that is, each record in the distributed task may be regarded as a monitoring point for monitoring consistency of the post-heterogeneous data and the pre-heterogeneous data, and the consistency may be embodied in terms of state information, time point, and the like.
Therefore, when monitoring a monitoring point generation event from a distributed task, the target service system may obtain a monitoring point corresponding to the monitoring point generation event, and may determine which service data in the original data source is heterogeneous to the target data source according to the monitoring point.
And S120, acquiring heterogeneous data corresponding to the monitoring point in the target data source, and verifying the heterogeneous data based on the monitoring point.
The data before the isomerism corresponding to the monitoring point in the original data source can be determined according to the monitoring point, and then the data after the isomerism corresponding to the monitoring point in the target data source can also be determined. There are various implementation manners for performing heterogeneous operation on the heterogeneous data according to the monitoring point, and at least two implementation manners of the various implementation manners may be arbitrarily combined. The following several alternatives are exemplified.
In an optional manner, whether the monitoring points (i.e., the newly added records corresponding to the monitoring points) are consistent with the heterogeneous data is verified based on a state machine verification task (i.e., worker verification), and when the newly added state information is consistent with the heterogeneous state information, the newly added records are consistent with the heterogeneous data, and certainly, the newly added records are inconsistent with the heterogeneous data if the newly added records are inconsistent with the heterogeneous state information, by combining with an application scenario that may be involved in an embodiment of the present invention. The status information may be in a warehouse, out of a warehouse, in a distribution, committed, rejected, etc., and the check scheme may be understood as a check scheme based on state machine consistency.
In an optional mode two, in some application scenarios, when the newly added state information and the heterogeneous state information are different, the data before and after the heterogeneous is also consistent. For example, when the newly added state information and the heterogeneous state information belong to different original service systems, the relative positions of the newly added state information and the heterogeneous state information in the preset appearance sequence of various state information can be determined, and the relative positions can indicate which of the two state information appears later in the whole service flow, so that the heterogeneous data is verified according to the relative positions. For example, if the whole business process is from first to last in warehouse-delivery, commissioned/rejected, the newly added status information belongs to warehouse-delivered in the warehousing system, and the heterogeneous status information belongs to distribution in the distribution system, commissioned or rejected, since in the whole business process of the e-commerce platform scenario, the commissioned, rejected are all in delivery after the warehouse-delivered, which means that when a certain commodity is in delivery, commissioned or rejected, it must be delivered, and then the heterogeneous data is consistent with the newly added record.
In an alternative, generally, service data needs a certain time to be able to be heterogeneous from an original data source to a target data source, it is difficult to be heterogeneous from the original data source to the target data source at once, such as in MQ, it is difficult for the target service system to consume a received MQ message to the target data source at once, but a monitoring point may be generated at the beginning of data heterogeneous. Therefore, in order to avoid the situation that data verification fails due to data verification when data isomerism is not completely executed, a preset time window may be configured in the monitoring point, and when the monitoring point generates a preset time window after the monitoring time point of the event, the heterogeneous data is verified based on the monitoring point, where the monitoring time point is the time point when the event generated by the monitoring point is monitored. In other words, the monitoring time point is taken as a starting time point, after a preset time window passes from the starting time point, the heterogeneous data is verified based on the monitoring point, that is, the distributed task is executed again after the preset time window to verify the heterogeneous data. Illustratively, the preset time window may be 10s, which means that the service data in the original data source is allowed to be asynchronously updated to the target data source within 10 s.
In an alternative, in some application scenarios, in addition to the state information in the service data, the time point of the service data needs to be considered, for example, in the business process of e-commerce platform scenario, a commodity can be transported from one site to another site, which is a delivery state, and the delivery state can be overridden by processing logic in the target business system, such as that the delivery of the commodity from site a to site B is a delivery state, while the transfer from station B to station C is another transfer state that may overlap each other, but a transfer state occurring later should not overlap a transfer state occurring earlier, such as the transfer of goods from station a to station B occurring at 14, and when the data is transmitted from the station B to the station C at the 15 points, the service data of the 14 points should not cover the service data of the 15 points, and at the moment, the time sequence check can be added when checking the heterogeneous data. Specifically, if the newly added state information and the heterogeneous state information are both preset state information such as a station transmission state, respectively acquiring a newly added time point when a monitoring point is newly added to a distributed task and a heterogeneous time point when heterogeneous data is heterogeneous to a target data source; and obtaining the current time point, comparing with the newly added time point, if the heterogeneous time point is closer to the current time point, namely the newly added time point is earlier and the heterogeneous time point is later, checking the heterogeneous data according to whether the time difference between the heterogeneous time point and the newly added time point is greater than a preset time threshold, and if the time difference is greater than the preset time threshold, indicating that the target service system has changed the heterogeneous data according to the service logic of the target service system, wherein the heterogeneous data can be considered to be consistent with the monitoring point.
And S130, if the data after the isomerism is determined to be inconsistent with the data before the isomerism corresponding to the monitoring point according to the checking result, updating the data after the isomerism according to the data before the isomerism.
After checking the heterogeneous data based on the monitoring point, a checking result of whether the monitoring point and the heterogeneous data are consistent can be obtained. Because the monitoring point and the data before the isomerism are consistent, whether the data after the isomerism is consistent with the data before the isomerism corresponding to the monitoring point or whether the service data in the original data source is consistent with the service data in the target data source or not can be determined according to the checking result. It should be noted that, if the two are inconsistent, this may be because the target service system changes the heterogeneous service data based on its own service logic, or the service data is not successfully heterogeneous from the original data source to the target data source, which may cause the heterogeneous service data to be inconsistent in the original data source and the target data source.
In order to ensure the consistency of the heterogeneous service data, the heterogeneous data can be updated according to the pre-heterogeneous data, for example, the pre-heterogeneous data corresponding to the monitoring point is obtained based on the monitoring point, and the heterogeneous data in the target data source is refreshed according to the pre-heterogeneous data, and this refresh operation can also be understood as a resynchronization operation executed based on the monitoring point, and this resynchronization mechanism ensures the robustness of the target service system. In practical application, optionally, when the target service system monitors that the original data source and the target data source are different, the target service system may send a resynchronization notification, so that a relevant module for performing resynchronization operation in the target service system performs resynchronization operation after receiving the resynchronization notification.
According to the technical scheme of the embodiment of the invention, when a target service system monitors a monitoring point generation event from a distributed task, a monitoring point corresponding to the monitoring point generation event can be obtained, namely a record newly added in the distributed task when service data are heterogeneous from an original data source to a target data source is obtained; and then, checking heterogeneous data corresponding to the monitoring point in a target data source based on the monitoring point, and if the heterogeneous data is determined to be inconsistent with the pre-heterogeneous data in the original data source corresponding to the monitoring point according to a checking result, updating the heterogeneous data according to the pre-heterogeneous data. According to the technical scheme, the heterogeneous service data in the target data source is verified through the corresponding monitoring points generated in the data heterogeneous process, and the data updating operation is executed when the service data is determined to be inconsistent in the two data sources according to the verification result, so that the consistency of the heterogeneous service data in the two data sources is guaranteed.
Example two
Fig. 2 is a flowchart of a data heterogeneous method according to a second embodiment of the present invention. The present embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the data heterogeneous method may further include: when a data updating event derived from a theme in the MQ is monitored, consuming the MQ according to the data updating event; writing the heterogeneous pre-data consumed from the MQ into a target data source, and generating a monitoring point in the distributed task according to a consumption result; correspondingly, updating the post-isomerism data according to the pre-isomerism data may include: consuming the MQ again to obtain data before isomerism; and updating the data after the isomerism according to the obtained data before the isomerism. The same or corresponding terms as those in the above embodiments are not explained in detail herein.
Referring to fig. 2, the method of the present embodiment may specifically include the following steps:
s210, when monitoring a data updating event derived from the theme in the MQ, consuming the MQ according to the data updating event, writing the pre-heterogeneous data consumed from the MQ into a target data source, and generating a monitoring point in the distributed task according to a consumption result.
The data heterogeneous scheme based on binlog consumption may have a problem of data synchronization delay, because the binlog consumption guarantees sequence consistency by adopting a serial consumption mode, specifically, the binlog consumption is a process of acquiring change operation logs from a master by the binlog, when a plurality of change operation logs exist at the same time, the binlog consumption needs to read all the change operation logs at the time after all the change operation logs at the time are completely read, so that the change operation logs at the next time can be read, which may cause data synchronization delay, and the data synchronization delay is also one of reasons for causing inconsistency of heterogeneous service data.
In order to solve the problem of data synchronization delay, the embodiment of the invention provides a scheme for performing data isomerism by adopting an MQ mode. Specifically, while writing the changed service data into the original data source, the original service system may also write the changed service data into an MQ, specifically, into a topic (topic) of the MQ, where the topic is a storage unit known by the original service system and the target service system, and the original service system is a sending end of the MQ. On the basis, the target service system can monitor the topic and consume the MQ when a data updating event occurs in the topic, namely acquiring heterogeneous pre-data corresponding to the data updating event from the topic of the MQ and writing the heterogeneous pre-data into a target data source, so that the execution efficiency of data heterogeneous is improved by tracking the parallel consumption of the MQ based on a service flow, and the target service system is a consuming end of the MQ. Meanwhile, monitoring points can be generated in the distributed tasks according to consumption results, so that heterogeneous service data can be verified according to the monitoring points in the following process.
S220, when monitoring that a monitoring point generation event from the distributed task is generated, acquiring a monitoring point corresponding to the monitoring point generation event, wherein the monitoring point is a record newly added in the distributed task when data before isomerization in an original data source is isomerized to a target data source.
And S230, acquiring heterogeneous data corresponding to the monitoring point in the target data source, and verifying the heterogeneous data based on the monitoring point.
And S240, if the fact that the data after the isomerism is inconsistent with the data before the isomerism corresponding to the monitoring point is determined according to the check result, consuming the MQ again to obtain the data before the isomerism, and updating the data after the isomerism according to the obtained data before the isomerism.
When the deviation of the service data in the target data source is determined according to the check result, the MQ can be consumed so as to obtain corresponding data before the isomerism from the MQ, and then the data after the isomerism is updated according to the obtained data before the isomerism.
According to the technical scheme of the embodiment of the invention, when the data updating event in the topic of the MQ is monitored, the MQ is consumed in parallel according to the data updating event, and the consumption result is written into the target data source, so that the problem of data synchronization delay is solved, and the data heterogeneous efficiency is improved.
An optional technical solution, writing heterogeneous pre-data consumed from the MQ into the target data source, may include: and doubly writing the heterogeneous pre-data consumed from the MQ into a first data table and a target data source, wherein the first data table stores the full information of each item. The reason for this is that when a user queries data, the query is mainly performed in a database with relatively high data query efficiency, such as ES, and therefore, heterogeneous pre-data consumed from the MQ can be doubly written into the first data table, which may be a data table stored in the database with relatively high data query efficiency. In other words, each piece of pre-heterogeneous data may be directly synchronized into the first data table, and the first data table may store the full amount information of each item, for example, in an e-commerce platform scenario, and the first data table may store the full amount information of the customer information, the ex-warehouse time, the delivery time, and the like of each order, so that the user may perform a full amount query in the first data table. Since the first data table contains more fields, such a data table may be referred to as a wide table.
In addition, when the target data source system is a service sub-base, the target data source system is not suitable for query of data report information, and when the target data source receives heterogeneous data, the target data source can further isomerize the heterogeneous data into a second data table again, for example, the target data source synchronously produces the second data table when receiving the binlog in the mysql where the target data source is located, and the second data table can also be a data table in a database with higher data query efficiency. However, when the target data source performs data isomerism through binlog consumption, as described above, since the binlog consumption is serial consumption, there is a problem of data synchronization delay, and in order to increase the data synchronization speed, data before isomerism may be double-written into the first data table and the target data source.
On the basis of any technical solution, optionally, in order to implement cold and hot data isolation of the first data table and the second data table and to make the first data table and the second data table be cold devices (the first data table stores cold data and the second data table stores hot data), the data heterogeneous method may further include: when a data change event from a target data source is monitored, determining heterogeneous state information in heterogeneous data corresponding to the data change event; if it is determined that the business process node corresponding to the heterogeneous data is not the business end node according to the heterogeneous state information, the heterogeneous data may be heterogeneous to the second data table, so that hot data is stored in the second data table, where the hot data may be understood as unfinished data, for example, in an e-commerce platform scenario, the unfinished data may be unfinished orders, such as unfinished orders, unfinished orders for delivery, and the like. In other words, in practical applications, it may be determined whether the heterogeneous data is an unfinished order according to the order status in the heterogeneous data, and the heterogeneous data belonging to the unfinished order is heterogeneous in the second data table.
It should be noted that, firstly, the purpose of heterogeneous configuration of the service data from the target data source to the second data table is that the query process of the mass data is more suitable for the database with faster data query efficiency, and because the amount of the hot data is smaller, for the second data table stored in the ES, the ES can directly put the hot data in the second data table into the memory for query, so that the query efficiency is higher. Secondly, when the target data source is mysql, the data heterogeneous process can adopt various modes such as binlog consumption and MQ, compared with other modes, the binlog consumption is more direct, the binlog can be directly consumed through the binlog, and business codes do not need to be additionally written in business logic, so that the complexity of the business logic is reduced; moreover, the data heterogeneous process from the target data source to the second data table, which is a completely equivalent replication process of the traffic data because the second data table itself has no iterative update due to no traffic logic, does not exist, which means that there is no data synchronization delay problem even if binlog consumption is adopted.
EXAMPLE III
Fig. 3 is a flowchart of a data heterogeneous method according to a third embodiment of the present invention. The present embodiment is optimized based on the above technical solutions. In this embodiment, optionally, the data heterogeneous method may further include: when a data sampling event is monitored, determining a data sampling condition according to the data sampling event; respectively acquiring a first sampling result in a target data source and a second sampling result in an existing data source according to a data sampling condition; and taking the first sampling result and the second sampling result as a group of verification sampling results, and verifying the correctness of each service data in the target data source according to the plurality of groups of verification sampling results. The same or corresponding terms as those in the above embodiments are not explained in detail herein.
Referring to fig. 3, the method of this embodiment may specifically include the following steps:
s310, when monitoring that a monitoring point generation event from the distributed task is generated, acquiring a monitoring point corresponding to the monitoring point generation event, wherein the monitoring point is a record newly added in the distributed task when data before isomerization in an original data source is isomerized to a target data source.
S320, acquiring heterogeneous data corresponding to the monitoring point in the target data source, and verifying the heterogeneous data based on the monitoring point; and if the data after the isomerism is determined to be inconsistent with the data before the isomerism corresponding to the monitoring point according to the checking result, updating the data after the isomerism according to the data before the isomerism.
S330, when a data sampling event is monitored, determining a data sampling condition according to the data sampling event, and respectively obtaining a first sampling result in a target data source and a second sampling result in an existing data source according to the data sampling condition.
In view of the application scenario that may be involved in the embodiment of the present invention, the target service system may be a service system obtained by upgrading an existing service system in which an existing data source is located, and in order to verify the correctness of each service data stored in the target service system, the target service system and the existing service system may run in parallel. When monitoring a data sampling event, the target service system can perform data sampling in a target data source according to a data sampling condition corresponding to the data sampling event to obtain a first sampling result; in addition, a second sampling result sent by the existing data source may be obtained, where the second sampling result may be a sampling result sampled by the existing service system in the existing data source according to the data sampling condition.
And S340, taking the first sampling result and the second sampling result as a group of verification sampling results, and verifying the correctness of each service data in the target data source according to the plurality of groups of verification sampling results.
And determining whether the service data corresponding to the data sampling condition in the target data source is correct or not according to the comparison result between the first sampling result and the second sampling result. Then, when the first sampling result and the second sampling result are used as a group of verification sampling results, whether each service data in the target data source is correct or not can be verified according to the multiple groups of verification sampling results, if the group of verification sampling results are consistent in comparison, the result can be recorded as a one-time pass, otherwise, the result can be recorded as a one-time mismatch, the correctness of each service data in the target data source can be verified subsequently according to the number of passes and the number of mismatches, the target service system can be improved when the correctness is low, and the sampling verification mechanism ensures the robustness of the target service system.
According to the technical scheme of the embodiment of the invention, the first sampling result and the second sampling result which are respectively obtained in the target data source and the existing data source according to the data sampling condition are compared, and the correctness of each service data in the target data source can be verified according to the comparison result of each group of sampling results, so that the robustness of the target service system is improved.
In order to better understand the specific implementation process of the above steps, the following continues to use the above e-commerce platform scenario as an example to exemplarily describe the data heterogeneous method of the present embodiment. Illustratively, as shown in fig. 4 a-4 c, the 3 figures are different parts of a schematic diagram of an architecture, which is too large to be clearly shown in the figures of the specification, and for this reason, the 3 figures are respectively cut from different angles for display, and overlapped parts exist in the 3 figures. Specifically, as shown in fig. 4a, the data source a is an original data source shared by the warehousing system and the delivery system, the data source B is a target data source in the reporting system, and after the warehousing system and/or the delivery system writes the business data into the MQ, the reporting system reads the business data from the MQ, double writes the read result into the data source B and the wide table B (i.e., the first data table), and generates the monitoring point in the distributed task. And carrying out consistency check on the service data in the data source B and the data source A based on the monitoring point, and sending out a resynchronization notice to automatically resynchronize when the check result is abnormal. In addition, since the full-scale information of an order (which includes only summary information) may not be included in the MQ, but the full-scale information of the order needs to be included in the data source B, the business information that is not included in the MQ can be directly obtained from the data source a. The service data in the data source B can also be synchronized to the wide table a (i.e. the second data table) for storing the hot data, and the forwarding of the wide table a can be triggered when the data source B is forwarded for 3 months, so that the problem that the query speed of the data in the wide table a is reduced due to the fact that the data volume under the same index is too large is avoided. Similarly, a 2 year hold-off is referred to in Wide Table B, which may be a hold-off triggered at a preset time to hold off the completed order 2 years ago. As shown in fig. 4B, the data source B may include two tables, i.e., an index table and a data table, which are two data reports obtained by data isomerism. As shown in fig. 4c, in the data report page, data query can be performed in the report system, the wide table B or the wide table a according to different data query conditions. In addition, in the new query monitoring (3) module, an online sampling result (i.e., a second sampling result) and a report system sampling result (i.e., a first sampling result) can be obtained according to the data sampling condition, and a comparison result record of the two sampling results can be used for verifying the correctness of each service data in the report system. In addition, when a field is newly added in the data report, whether the newly added field is a query condition or not can be determined, if so, the newly added field can be added into a main table (data _ table1_ wide _ report) of the data report, otherwise, the newly added field can be added into an extension table (data _ table1_ wide _ extend _ report) of the data report, the extension table is a value-added service, wherein the number of the fields is small, and the newly added field is quicker in a massive data scene.
Example four
Fig. 5 is a block diagram of a data heterogeneous apparatus according to a fourth embodiment of the present invention, where the apparatus is configured to execute a data heterogeneous method according to any of the foregoing embodiments. The device and the data heterogeneous method of each embodiment belong to the same inventive concept, and details which are not described in detail in the embodiment of the data heterogeneous device may refer to the embodiment of the data heterogeneous method. Referring to fig. 5, the apparatus is configured in a target service system, where the target service system is provided with a target data source and a distributed task, and the apparatus may specifically include: a monitoring point acquisition module 410, a data verification module 420, and a data update module 430.
The monitoring point obtaining module 410 is configured to obtain a monitoring point corresponding to a monitoring point generation event when monitoring that the monitoring point generation event in the distributed task is originated, where the monitoring point is a record newly added in the distributed task when data before isomerization in an original data source is isomerized to a target data source;
the data verification module 420 is configured to obtain heterogeneous data corresponding to the monitoring point in the target data source, and verify the heterogeneous data based on the monitoring point;
and a data updating module 430, configured to update the data after the isomerization according to the data before the isomerization if it is determined that the data after the isomerization is inconsistent with the data before the isomerization corresponding to the monitoring point according to the check result.
Optionally, the data checking module 420 may specifically include:
and the first data verification unit is used for the monitoring point to be configured with a preset time window, and when the monitoring point generates the preset time window after the monitoring time point of the event, the heterogeneous data is verified based on the monitoring point.
Optionally, the data checking module 420 may specifically include:
the second data verification unit is used for respectively acquiring a newly added time point of the monitoring point newly added to the distributed task and an isomeric time point of the isomeric data to the target data source if the newly added state information in the monitoring point and the isomeric state information in the isomeric data are both preset state information; and acquiring a current time point, and comparing with the newly added time point, if the heterogeneous time point is closer to the current time point, checking the heterogeneous data according to whether the time difference value between the heterogeneous time point and the newly added time point is greater than a preset time threshold value.
Optionally, the data checking module 420 may specifically include:
the third data checking unit is used for acquiring the preset appearance sequence of each state information; and if the newly added state information in the monitoring point and the heterogeneous state information in the heterogeneous data belong to different original service systems, checking the heterogeneous data according to the relative positions of the newly added state information and the heterogeneous state information in the sequence of occurrence in the first and the last steps.
Optionally, on the basis of the above apparatus, the apparatus may further include:
the data heterogeneous module is used for consuming the MQ according to the data updating event when the data updating event derived from the theme in the MQ is monitored; writing the heterogeneous pre-data consumed from the MQ into a target data source, and generating a monitoring point in the distributed task according to a consumption result;
correspondingly, the data updating module 430 may specifically include:
the data updating unit is used for consuming the MQ again to obtain data before isomerism; and updating the data after the isomerism according to the obtained data before the isomerism.
Optionally, the data heterogeneous module may specifically include:
and the data double-writing unit is used for double-writing the heterogeneous pre-data consumed from the MQ into a first data table and a target data source, wherein the first data table stores the full information of each article.
Optionally, on the basis of the above apparatus, the apparatus may further include:
the state information determining module is used for determining heterogeneous state information in heterogeneous data corresponding to a data change event when the data change event from a target data source is monitored;
and the second isomerization module is used for isomerizing the isomerized data into the second data table if the business process node corresponding to the isomerized data is determined not to be the business end node according to the isomerization state information.
Optionally, on the basis of the above apparatus, the apparatus may further include:
the sampling result obtaining module is used for determining data sampling conditions according to the data sampling events when the data sampling events are monitored; respectively acquiring a first sampling result in a target data source and a second sampling result in an existing data source according to a data sampling condition;
and the target data source verification module is used for taking the first sampling result and the second sampling result as a group of verification sampling results and verifying the correctness of each service data in the target data source according to the plurality of groups of verification sampling results.
Optionally, the original business system where the original data source is located includes a warehousing system and/or a distribution system, and the target business system includes a reporting system.
In the data heterogeneous device provided by the fourth embodiment of the present invention, when monitoring a monitoring point generation event from a distributed task, a monitoring point acquisition module may acquire a monitoring point corresponding to the monitoring point generation event, that is, acquire a record newly added to the distributed task when service data is heterogeneous from an original data source to a target data source; and then, the data checking module and the data updating module are mutually matched, heterogeneous data corresponding to the monitoring point in the target data source can be checked based on the monitoring point, and if the heterogeneous data is determined to be inconsistent with the heterogeneous data in the original data source corresponding to the monitoring point according to the checking result, the heterogeneous data can be updated according to the heterogeneous data. According to the device, the heterogeneous service data in the target data source is verified through the corresponding monitoring points generated in the data heterogeneous process, and the data updating operation is executed when the service data is determined to be inconsistent in the two data sources according to the verification result, so that the consistency of the heterogeneous service data in the two data sources is guaranteed.
The data heterogeneous device provided by the embodiment of the invention can execute the data heterogeneous method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
It should be noted that, in the embodiment of the data heterogeneous apparatus, each included unit and each included module are only divided according to functional logic, but are not limited to the above division as long as the corresponding function can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
EXAMPLE five
Fig. 6 is a schematic structural diagram of a target service system according to a fifth embodiment of the present invention, and as shown in fig. 6, the target service system includes a memory 510, a processor 520, an input device 530, and an output device 540. The number of processors 520 in the target business system may be one or more, and one processor 520 is taken as an example in fig. 6; the memory 510, processor 520, input device 530, and output device 540 in the target business system may be connected by a bus or other means, such as by bus 550 in fig. 6.
The memory 510 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the data heterogeneous method in the embodiment of the present invention (for example, the monitoring point obtaining module 410, the data checking module 420, and the data updating module 430 in the data heterogeneous device). The processor 520 executes various functional applications of the target business system and data processing by executing software programs, instructions and modules stored in the memory 510, that is, implements the data heterogeneous method described above.
The memory 510 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the target business system, and the like. Further, the memory 510 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 510 may further include memory located remotely from processor 520, which may be connected to devices through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 530 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the device. The output device 540 may include a display device such as a display screen.
EXAMPLE six
An embodiment of the present invention provides a storage medium containing computer-executable instructions, where the computer-executable instructions are used to execute a data heterogeneous method when executed by a computer processor, and the method is applied to a target service system, where the target service system is provided with a target data source and distributed tasks, and the method may include:
when monitoring a monitoring point generation event from a distributed task, acquiring a monitoring point corresponding to the monitoring point generation event, wherein the monitoring point is a record newly added in the distributed task when data in an original data source is heterogeneous to a target data source before the data is heterogeneous; acquiring heterogeneous data corresponding to the monitoring point in the target data source, and verifying the heterogeneous data based on the monitoring point; and if the data after the isomerism is determined to be inconsistent with the data before the isomerism corresponding to the monitoring point according to the checking result, updating the data after the isomerism according to the data before the isomerism.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the method operations described above, and may also perform related operations in the data heterogeneous method provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. With this understanding, the technical solutions of the present invention may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
The foregoing is considered as illustrative of the preferred embodiments of the invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (12)
1. A data heterogeneous method is applied to a target service system, wherein a target data source and a distributed task are arranged in the target service system, and the method comprises the following steps:
when monitoring a monitoring point generation event from the distributed task, acquiring a monitoring point corresponding to the monitoring point generation event, wherein the monitoring point is a record newly added in the distributed task when data before isomerization in an original data source is isomerized to the target data source;
acquiring heterogeneous data corresponding to the monitoring point in the target data source, and verifying the heterogeneous data based on the monitoring point;
and if the fact that the data after the isomerism is inconsistent with the data before the isomerism corresponding to the monitoring point is determined according to the checking result, updating the data after the isomerism according to the data before the isomerism.
2. The method according to claim 1, wherein the monitoring point is configured with a preset time window, and the checking the heterogeneous data based on the monitoring point comprises:
and when the monitoring point generates the preset time window after the monitoring time point of the event, checking the heterogeneous data based on the monitoring point.
3. The method of claim 1, wherein the checking the heterogeneous data based on the monitoring point comprises:
if the newly added state information in the monitoring point and the heterogeneous state information in the heterogeneous data are both preset state information, respectively acquiring a newly added time point when the monitoring point is newly added to the distributed task and a heterogeneous time point when the heterogeneous data is heterogeneous to the target data source;
and acquiring a current time point, and comparing with the newly added time point, if the heterogeneous time point is closer to the current time point, checking the heterogeneous data according to whether a time difference value between the heterogeneous time point and the newly added time point is greater than a preset time threshold value.
4. The method of claim 1, wherein the checking the heterogeneous data based on the monitoring point comprises:
acquiring the preset appearance sequence of each state information;
and if the newly added state information in the monitoring point and the heterogeneous state information in the heterogeneous data belong to different original service systems, verifying the heterogeneous data according to the relative positions of the newly added state information and the heterogeneous state information in the sequence.
5. The method of claim 1, further comprising:
when a data update event derived from a topic in an MQ is monitored, consuming the MQ according to the data update event;
writing the heterogeneous pre-data consumed from the MQ into the target data source, and generating the monitoring point in the distributed task according to a consumption result;
the updating the post-isomerism data according to the pre-isomerism data includes:
consuming the MQ again to obtain the data before isomerism;
and updating the data after the isomerism according to the obtained data before the isomerism.
6. The method as recited in claim 5, wherein the writing the pre-heterogeneous data consumed from the MQ to the target data source comprises:
doubly writing the pre-heterogeneous data consumed from the MQ to a first data table and the target data source, wherein the first data table stores full information for each item.
7. The method of claim 1, further comprising:
when a data change event from the target data source is monitored, determining heterogeneous state information in the heterogeneous data corresponding to the data change event;
and if the business process node corresponding to the heterogeneous data is determined not to be a business end node according to the heterogeneous state information, isomerizing the heterogeneous data into a second data table.
8. The method of claim 1, further comprising:
when a data sampling event is monitored, determining a data sampling condition according to the data sampling event;
respectively acquiring a first sampling result in the target data source and a second sampling result in the existing data source according to the data sampling conditions;
and taking the first sampling result and the second sampling result as a group of verification sampling results, and verifying the correctness of each service data in the target data source according to the plurality of groups of verification sampling results.
9. The method according to claim 1, wherein the original business system where the original data source is located comprises a warehousing system and/or a distribution system, and the target business system comprises a reporting system.
10. A data heterogeneous device configured in a target service system, where a target data source and a distributed task are provided in the target service system, the device comprising:
a monitoring point acquisition module, configured to acquire a monitoring point corresponding to a monitoring point generation event when monitoring the monitoring point generation event in the distributed task, where the monitoring point is a record newly added to the distributed task when data before isomerization in an original data source is isomerized to the target data source;
the data verification module is used for acquiring heterogeneous data corresponding to the monitoring point in the target data source and verifying the heterogeneous data based on the monitoring point;
and the data updating module is used for updating the data after the isomerism according to the data before the isomerism if the data after the isomerism is determined to be inconsistent with the data before the isomerism corresponding to the monitoring point according to the checking result.
11. A targeted business system, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a data heterogeneous method as recited in any one of claims 1-9.
12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a data isomerisation method according to any one of the claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011165198.0A CN113760923B (en) | 2020-10-27 | 2020-10-27 | Data heterogeneous method, device, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011165198.0A CN113760923B (en) | 2020-10-27 | 2020-10-27 | Data heterogeneous method, device, system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113760923A true CN113760923A (en) | 2021-12-07 |
| CN113760923B CN113760923B (en) | 2023-09-05 |
Family
ID=78785898
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011165198.0A Active CN113760923B (en) | 2020-10-27 | 2020-10-27 | Data heterogeneous method, device, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113760923B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103984715A (en) * | 2014-05-08 | 2014-08-13 | 武汉库百网络技术有限公司 | Data synchronizing and checking method, device and system of isomerous database |
| US20170351585A1 (en) * | 2016-06-03 | 2017-12-07 | International Business Machines Corporation | Transaction consistency query support for replicated data from recovery log to external data stores |
| CN108399256A (en) * | 2018-03-06 | 2018-08-14 | 北京慧萌信安软件技术有限公司 | Heterogeneous database content synchronization method, device and middleware |
| US20180232407A1 (en) * | 2017-02-10 | 2018-08-16 | Wipro Limited | Method and system for assessing quality of incremental heterogeneous data |
| CN110347746A (en) * | 2019-06-13 | 2019-10-18 | 武汉达梦数据库有限公司 | A kind of heterogeneous database synchrodata consistency desired result method and device |
| CN110413629A (en) * | 2019-08-06 | 2019-11-05 | 上海德拓信息技术股份有限公司 | The inspection method of heterogeneous database data consistency based on log parsing |
-
2020
- 2020-10-27 CN CN202011165198.0A patent/CN113760923B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103984715A (en) * | 2014-05-08 | 2014-08-13 | 武汉库百网络技术有限公司 | Data synchronizing and checking method, device and system of isomerous database |
| US20170351585A1 (en) * | 2016-06-03 | 2017-12-07 | International Business Machines Corporation | Transaction consistency query support for replicated data from recovery log to external data stores |
| US20180232407A1 (en) * | 2017-02-10 | 2018-08-16 | Wipro Limited | Method and system for assessing quality of incremental heterogeneous data |
| CN108399256A (en) * | 2018-03-06 | 2018-08-14 | 北京慧萌信安软件技术有限公司 | Heterogeneous database content synchronization method, device and middleware |
| CN110347746A (en) * | 2019-06-13 | 2019-10-18 | 武汉达梦数据库有限公司 | A kind of heterogeneous database synchrodata consistency desired result method and device |
| CN110413629A (en) * | 2019-08-06 | 2019-11-05 | 上海德拓信息技术股份有限公司 | The inspection method of heterogeneous database data consistency based on log parsing |
Non-Patent Citations (1)
| Title |
|---|
| 沈鑫 等: "基于多源异构数据聚合技术的电力物联网设备全景 信息构架模型研究与应用", 南方电网技术, vol. 10, no. 2 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113760923B (en) | 2023-09-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107958010B (en) | Method and system for online data migration | |
| CN108874558B (en) | Message subscription method of distributed transaction, electronic device and readable storage medium | |
| CN110795503A (en) | Multi-cluster data synchronization method and related device of distributed storage system | |
| CN112069265B (en) | Synchronization method of configuration data, business data system, computer system and medium | |
| WO2016127903A1 (en) | Data synchronization method, device and system | |
| CN108874567B (en) | Service processing method and system | |
| US10747776B2 (en) | Replication control using eventually consistent meta-data | |
| US20160294651A1 (en) | Method, apparatus, and computer program product for monitoring an electronic data exchange | |
| CN110858197A (en) | Method and device for synchronizing data | |
| US10397125B2 (en) | Method of cross-regional data transmission and system thereof | |
| US20180121532A1 (en) | Data table partitioning management method and apparatus | |
| US11042454B1 (en) | Restoration of a data source | |
| CN111159300A (en) | Data processing method and device based on block chain | |
| CN114416868A (en) | Data synchronization method, device, equipment and storage medium | |
| CN113760923B (en) | Data heterogeneous method, device, system and storage medium | |
| CN117336135A (en) | Data transmission method, device, storage medium and electronic equipment | |
| CN112860746B (en) | Cache reduction-based method, equipment and system | |
| CN114661693A (en) | Data auditing realization method, storage medium, electronic equipment and system | |
| US11113311B2 (en) | Technology agnostic system and method for achieving eventually-consistent data replication | |
| CN113760845A (en) | Log processing method, system, device, client and storage medium | |
| US10997216B1 (en) | Systems and methods for centralized database cluster management | |
| CN113934792A (en) | Processing method and device of distributed database, network equipment and storage medium | |
| CN117349384B (en) | Database synchronization method, system and equipment | |
| CN113792891B (en) | Machine learning feature production system and method | |
| TWI461944B (en) | Electrical data synchronization method and electrical data synchronization system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |