CN117591607B - Data synchronization management method and system - Google Patents
Data synchronization management method and system Download PDFInfo
- Publication number
- CN117591607B CN117591607B CN202410077035.9A CN202410077035A CN117591607B CN 117591607 B CN117591607 B CN 117591607B CN 202410077035 A CN202410077035 A CN 202410077035A CN 117591607 B CN117591607 B CN 117591607B
- Authority
- CN
- China
- Prior art keywords
- data
- service
- service system
- determining
- reliability
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000007726 management method Methods 0.000 title claims abstract description 57
- 230000001360 synchronised effect Effects 0.000 claims abstract description 144
- 238000012545 processing Methods 0.000 claims abstract description 92
- 238000000034 method Methods 0.000 claims description 36
- 238000000605 extraction Methods 0.000 claims description 17
- 230000005856 abnormality Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 6
- 238000013075 data extraction Methods 0.000 claims description 5
- 230000002159 abnormal effect Effects 0.000 description 10
- 238000012216 screening Methods 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9035—Filtering based on additional data, e.g. user or group profiles
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Hardware Redundancy (AREA)
Abstract
The invention provides a data synchronization management method and a system, which belong to the technical field of data processing and specifically comprise the following steps: and determining the synchronous processing sequence of different synchronous service systems according to the association relation between the historical data reliability of the different synchronous service systems and the service data in the reference service system, and carrying out synchronous processing of the service data of the service systems based on the synchronous processing sequence until the service data of all the service systems are synchronously processed, and carrying out unified display on the service data of the different service systems based on the service requirements of users, thereby improving the accuracy and reliability of the data synchronous processing.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to a data synchronization management method and system.
Background
For users of the same type, a software service provider often provides a plurality of service systems meeting various service requirements, for example, school users, and needs to provide a plurality of service systems including campus card management, restaurant management, monitoring management, financial integrated management, etc., and as a certain correlation exists between data of different service systems, how to realize synchronous management of data of different service systems, and on the basis of ensuring the data quality of the service systems, the user can more accurately grasp the operation condition of the real service systems becomes a technical problem to be solved.
In order to solve the above technical problems, in the prior art, in the invention patent CN202210854436.1, "data synchronization method, system, device and storage medium", data processing is performed on metadata of different service systems based on a data structure of a target system, and after the metadata is adapted to the target system, target data is written into the target system through a data engine, but the following technical problems exist:
For different service systems needing to perform synchronous processing, on one hand, the complexity of the synchronous processing is different to a certain extent due to the difference of data interfaces of the different service systems, and on the other hand, the data authenticity of the same service data between the different service systems is possibly different to a certain extent, so that if the synchronous processing of the different service systems cannot be performed step by step, the efficiency of the synchronous processing of the data is difficult to meet the requirement.
The invention provides a data synchronization management method and a system for solving the technical problems.
Disclosure of Invention
In order to achieve the purpose of the invention, the invention adopts the following technical scheme:
According to one aspect of the present invention, a data synchronization management method is provided.
The data synchronization management method is characterized by comprising the following steps of:
S1, utilizing abnormal data conditions of different business data in different business systems during history synchronous processing to determine the reliability of the history data of the different business systems and the reliable business systems;
S2, determining data sources corresponding to different service data based on the types of the service data of different service systems to be synchronously processed, acquiring the number of the data sources in different reliable service systems, and determining the data reliability of different reliable service systems and the reference service system by combining the associated other service systems of the service data corresponding to the different data sources and the historical data reliability;
S3, taking the service system except the reference service system as a synchronous service system, determining the extraction difficulty of service data of different synchronous service systems according to the data architecture of different synchronous service systems, and determining the synchronous processing sequence of different synchronous service systems by combining the correlation relationship between the historical data reliability of different synchronous service systems and the service data in the reference service system;
And S4, carrying out synchronous processing on the service data of the service systems based on the synchronous processing sequence until the synchronous processing of the service data of all the service systems is completed, and uniformly displaying the service data of different service systems based on the service requirements of users.
The invention has the beneficial effects that:
1. The method and the device utilize the abnormal data conditions of different service data in the service system in the history synchronous processing to determine the reliable service system, realize the screening of the reliable service system with less update failure times and less times of non-update success times, and lay a foundation for the screening of the reference service system with higher reliability.
2. The method comprises the steps of determining a reference service system according to the number of data sources, other service systems associated with service data corresponding to different data sources and the reliability of historical data, singly considering the reliability of the historical data of the reference service system, and screening the service systems with more data sources by further considering the distribution condition of the data sources in the service system, so that the reliability of the use of the reference service system is ensured.
3. Based on the extraction difficulty, the correlation relation between the historical data reliability and the service data in the reference service system, the synchronous processing sequence of different synchronous service systems is determined, the determination of the synchronous processing sequence of the synchronous service systems with the data reliability, the extraction difficulty and the correlation condition with the reference service system differentiated from three angles is realized, and the synchronous processing efficiency is ensured.
The further technical scheme is that the data abnormal condition comprises data abnormal types and data abnormal times of different data abnormal types.
The further technical scheme is that the data exception type comprises unexploited and data updating exception.
The further technical scheme is that the method for determining the extraction difficulty of the service data of the synchronous service system comprises the following steps:
And determining a data protocol of the synchronous service system according to the data architecture of the synchronous service system, and determining the extraction difficulty of service data of the synchronous service system according to the data protocol.
The further technical scheme is that the method for determining the synchronous processing sequence of the synchronous service system comprises the following steps:
determining the association quantity of different synchronous service systems and service data in the reference service system according to the association relation between the different synchronous service systems and the service data in the reference service system, and determining the service association of the synchronous service systems and the reference service system by combining the quantity of data sources in the associated service data;
and determining a processing priority value of the synchronous service system based on the data extraction difficulty, the historical data reliability and the service relevance, and determining a synchronous processing sequence of the synchronous service system based on the processing priority value.
In a second aspect, the present invention provides a computer system comprising: a communicatively coupled memory and processor, and a computer program stored on the memory and capable of running on the processor, characterized by: the processor executes a data synchronization management method as described above when running the computer program.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the invention as set forth hereinafter.
In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
The above and other features and advantages of the present invention will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings;
FIG. 1 is a flow chart of a method of data synchronization management;
FIG. 2 is a flow chart of a method of determination of a reliable service system;
FIG. 3 is a flow chart of a method of determining a reference business system;
FIG. 4 is a flow chart of a method of determining a synchronization processing order of the synchronous business system;
FIG. 5 is a block diagram of a computer system.
Detailed Description
In order to make the technical solutions in the present specification better understood by those skilled in the art, the technical solutions in the embodiments of the present specification will be clearly and completely described below with reference to the drawings in the embodiments of the present specification, and it is obvious that the described embodiments are only some embodiments of the present specification, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present disclosure.
For software service enterprises, corresponding service systems exist for different service scenes of specific clients, for example, for school users, various service systems such as campus card management, restaurant management, monitoring management, financial integrated management and the like are often included, so that how to realize data synchronous management among different service systems becomes a technical problem to be solved urgently.
In order to solve the technical problems, the following technical scheme is adopted:
Firstly, determining the historical data reliability and the reliable service system of different service systems by utilizing the data anomaly condition of different service data in different service systems during the historical synchronous processing, in one embodiment, determining the historical data reliability of the service systems according to the frequency ratio of the abnormal data existing in the service systems during the historical synchronous processing, and taking the service system with higher historical data reliability as the reliable service system;
Then determining data sources corresponding to different service data based on the types of the service data of different service systems to be synchronously processed, specifically illustrating, for example, consumption data of a campus card in a canteen, wherein the data sources exist in a dining management system and in a campus card management and financial integrated management system, so that the data reliability of different reliable service systems and the determination of a reference service system can be performed through the number of the data sources in the reliable service systems, the associated other service systems of the service data corresponding to the different data sources and the historical data reliability, the determination of the weight value of the different data sources can be performed through the number of the associated other service systems of the service data corresponding to the different data sources, the determination of the data source reliability of the reliable service system is performed according to the normalized sum of the weight values of the data sources, and the service system with the maximum data reliability is used as the reference service system based on the sum of the data source reliability and the historical data reliability;
Then the service system except the reference service system is used as a synchronous service system, the extraction difficulty of service data of different synchronous service systems is determined according to the data architecture of different synchronous service systems, the data architecture of the campus card management and financial integrated management system is different, the storage protocol mode of data storage of the campus card management and financial integrated management system is different, the extraction difficulty can be determined by matching the storage protocol, and the synchronous processing sequence of different synchronous service systems is determined by combining the correlation relation between the historical data reliability of different synchronous service systems and the service data in the reference service system;
And finally, carrying out synchronous processing on the service data of the service systems based on the synchronous processing sequence until the synchronous processing of the service data of all the service systems is completed, and uniformly displaying the service data of different service systems based on the service requirements of users.
In order to solve the above-mentioned problems, according to one aspect of the present invention, as shown in fig. 1, there is provided a data synchronization management method, which is characterized by specifically comprising:
S1, utilizing abnormal data conditions of different business data in different business systems during history synchronous processing to determine the reliability of the history data of the different business systems and the reliable business systems;
specifically, the data anomaly condition includes data anomaly types and data anomaly times of different data anomaly types.
Further, the data exception types include unexploited, data update exceptions.
In a possible embodiment, as shown in fig. 2, the method for determining the reliable service system in the step S1 is:
Determining the historical data reliability of different business data of a business system based on the data anomaly types of different business data of the business system and the data anomaly times of different data anomaly types, and determining the system data reliability of the business system by combining the number of business systems associated with different business data;
Determining the synchronization processing reliability of the service system according to the quantity ratio of service data with data abnormality and the data abnormality type of the service system under different historical synchronization processing times, and determining the synchronization reliability of the service system by combining the moments corresponding to the different historical synchronization processing times;
And determining the historical data reliability of the reliable service system according to the system data reliability and the synchronous reliability of the service system, and determining whether the service system is the reliable service system or not according to the historical data reliability.
It can be appreciated that when the historical data reliability of the business system is greater than a preset reliability threshold, then the business system is determined to be a reliable business system.
In another possible embodiment, the method for determining the reliable service system in the step S1 is:
Determining the historical data reliability of different business data of a business system based on the data anomaly types of different business data of the business system and the data anomaly times of different data anomaly types, determining the system data reliability of the business system by combining the number of business systems associated with different business data, judging whether the system data reliability of the business system does not meet the requirement, if so, determining that the business system does not belong to a reliable business system, and determining the historical data reliability of the business system through the system data reliability;
determining the synchronization processing reliability of a service system under different historical synchronization processing times according to the quantity proportion of service data with data anomalies and the data anomaly types of the service system under different historical synchronization processing times, determining the synchronization reliability of the service system according to the time corresponding to the different historical synchronization processing times, judging whether the synchronization reliability of the service system does not meet the requirement, if so, determining that the service system does not belong to a reliable service system, and determining the historical data reliability of the service system according to the synchronization reliability;
And determining the historical data reliability of the reliable service system according to the system data reliability and the synchronous reliability of the service system, and determining whether the service system is the reliable service system or not according to the historical data reliability.
In another possible embodiment, the method for determining the reliable service system in the step S1 is:
Determining weight values of different business data according to the number of business systems associated with different business data, determining the historical data reliability of the different business data of the business system based on the data exception types of the different business data of the business system and the data exception times of the different data exception types, judging whether the sum of the weight values of the business data with the historical data reliability not meeting the requirements meets the requirements, if so, entering the next step, if not, determining that the business system does not belong to a reliable business system, and determining the historical data reliability of the business system through the sum of the weight values of the business data with the historical data reliability not meeting the requirements;
Determining the weight value of different historical synchronization times based on the moment corresponding to the different historical synchronization times, determining the synchronization reliability of the service system under different historical synchronization times according to the number proportion of service data with data anomalies and the data anomaly types of the service system under the different historical synchronization times, judging whether the sum of the weight values of the historical synchronization times with the synchronization reliability not meeting the requirement meets the requirement, if yes, entering the next step, if no, determining that the service system does not belong to a reliable service system, and determining the historical data reliability of the service system according to the sum of the weight values of the historical synchronization times with the synchronization reliability not meeting the requirement;
Acquiring the sum of weight values of service data of which the reliability of the historical data does not meet the requirement, determining the reliability of system data of the service system according to the historical data reliability of different service data of the service system and the weight values of different service data, judging whether the reliability of the system data of the service system does not meet the requirement, if so, determining that the service system does not belong to a reliable service system, and determining the reliability of the historical data of the service system through the reliability of the system data;
Acquiring the sum of weight values of historical synchronization processing times of which the synchronization processing reliability of the service system does not meet the requirement, determining the synchronization reliability of the service system by combining the synchronization processing reliability and the weight values of different historical synchronization processing times, judging whether the synchronization reliability of the service system does not meet the requirement, if so, determining that the service system does not belong to a reliable service system, and determining the historical data reliability of the service system through the synchronization reliability;
And determining the historical data reliability of the reliable service system according to the system data reliability and the synchronous reliability of the service system, and determining whether the service system is the reliable service system or not according to the historical data reliability.
S2, determining data sources corresponding to different service data based on the types of the service data of different service systems to be synchronously processed, acquiring the number of the data sources in different reliable service systems, and determining the data reliability of different reliable service systems and the reference service system by combining the associated other service systems of the service data corresponding to the different data sources and the historical data reliability;
In one possible embodiment, as shown in fig. 3, the method for determining the reference service system in the step S2 is as follows:
Determining the data source reliability of the reliable service system according to the number of other associated service systems of the service data corresponding to different data sources and the number of the data sources;
And determining the data reliability of the reliable service system based on the data source reliability and the historical data reliability, and determining whether the reliable service system belongs to a reference service system or not according to the data reliability.
In another possible embodiment, the method for determining the reference service system in the step S2 is:
determining weight values of different data sources according to the number of other associated service systems of service data corresponding to the different data sources, and dividing the data sources into associated data sources and general data sources based on the weight values of the data sources;
Determining the reliability of the data sources of the associated data sources of the reliable service system according to the number of the associated data sources of the reliable service system and the weight values of different associated data sources, and determining the reliability of the data sources of the general data sources of the reliable service system according to the number of the general data sources of the reliable service system and the weight values of different general data sources;
And determining the data reliability of the reliable service system based on the data source reliability of the general data source, the data source reliability of the related data source and the historical data reliability, and determining whether the reliable service system belongs to a reference service system or not through the data reliability.
In another possible embodiment, the method for determining the reference service system in the step S2 is:
When the number of data sources in the reliable service system is smaller than the preset number of data sources and the historical data reliability threshold of the reliable service system is smaller than the preset reliability threshold: determining that the reliable service system does not belong to a reference service system;
determining weight values of different data sources according to the number of other associated service systems of service data corresponding to the different data sources, and dividing the data sources into associated data sources and general data sources based on the weight values of the data sources;
when the number of the data sources in the reliable service system is smaller than the preset number of the data sources or the historical data reliability threshold of the reliable service system is smaller than the preset reliability threshold:
Determining the reliability of the data sources of the associated data sources of the reliable service system according to the number of the associated data sources of the reliable service system and the weight values of different associated data sources, judging whether the reliability of the data sources of the associated data sources meets the requirement, if so, determining that the service system does not belong to a reference service system, and if not, entering the next step;
When the number of the data sources in the reliable service system is not less than the preset number of the data sources and the historical data reliability threshold of the reliable service system is not less than the preset reliability threshold:
And determining the data source reliability of the general data source of the reliable service system according to the number of the general data sources of the reliable service system and the weight values of different general data sources, determining the data reliability of the reliable service system based on the data source reliability of the general data source, the data source reliability of the related data source and the historical data reliability, and determining whether the reliable service system belongs to a reference service system according to the data reliability.
In another possible embodiment, the method for determining the reference service system in the step S2 is:
S21, judging whether the historical data reliability threshold of the reliable service system is smaller than a preset reliability threshold, if so, entering the next step, and if not, entering the step S24;
S22, judging whether the number of the data sources of the reliable service system is smaller than the number of the preset data sources, if so, determining that the reliable service system does not belong to a reference service system, and if not, entering step S23;
S23, determining the weight value of different data sources according to the number of other associated service systems of service data corresponding to the different data sources, dividing the data sources into associated data sources and general data sources based on the weight values of the data sources, determining the data source reliability of the associated data sources of the reliable service system according to the number of the associated data sources of the reliable service system and the weight values of the different associated data sources, judging whether the data source reliability of the associated data sources meets the requirement, if yes, determining that the service system does not belong to a reference service system, and if no, entering the next step;
s24, determining the data source reliability of the general data source of the reliable service system according to the number of the general data sources of the reliable service system and the weight values of different general data sources, determining the data reliability of the reliable service system according to the data source reliability of the general data source, the data source reliability of the related data source and the historical data reliability, and determining whether the reliable service system belongs to a reference service system according to the data reliability.
S3, taking the service system except the reference service system as a synchronous service system, determining the extraction difficulty of service data of different synchronous service systems according to the data architecture of different synchronous service systems, and determining the synchronous processing sequence of different synchronous service systems by combining the correlation relationship between the historical data reliability of different synchronous service systems and the service data in the reference service system;
In one possible embodiment, the method for determining the extraction difficulty of the service data of the synchronous service system in the step S3 includes:
And determining a data protocol of the synchronous service system according to the data architecture of the synchronous service system, and determining the extraction difficulty of service data of the synchronous service system according to the data protocol.
In one possible embodiment, as shown in fig. 4, the method for determining the synchronization processing sequence of the synchronous service system in the step S3 is as follows:
S31, determining the association quantity of different synchronous service systems and service data in the reference service system through the association relation between the different synchronous service systems and the service data in the reference service system, and determining the service association of the synchronous service systems and the reference service system by combining the quantity of data sources in the associated service data;
S32, determining a processing priority value of the synchronous service system based on the data extraction difficulty, the historical data reliability and the service relevance, and determining the synchronous processing sequence of the synchronous service system based on the processing priority value.
In another possible embodiment, the method for determining the synchronization processing sequence of the synchronous service system in the step S3 is as follows:
determining the association quantity of different synchronous service systems and service data in the reference service system according to the association relation between the different synchronous service systems and the service data in the reference service system, and determining the service association of the synchronous service systems and the reference service system by combining the quantity of data sources in the associated service data;
And determining the processing priority of the synchronous service system through the historical data processing reliability of the synchronous service system, and determining the processing priority value of the synchronous service system based on the data extraction difficulty and the service relevance when the processing priority of the synchronous service system is the same, and determining the synchronous processing sequence of the synchronous service system based on the processing priority value.
And S4, carrying out synchronous processing on the service data of the service systems based on the synchronous processing sequence until the synchronous processing of the service data of all the service systems is completed, and uniformly displaying the service data of different service systems based on the service requirements of users.
Specifically, the method for displaying the service data of different service systems in a unified manner based on the service requirements of the users specifically includes:
and determining a matched service system of the service requirement based on the service requirement of the user, and determining the service data of uniform display according to the service data of the matched service system.
The following will be further described in connection with one specific scenario:
Firstly, determining the reliability of historical data of a service system by utilizing the frequency proportion of abnormal data existing in the history synchronous processing of different financial integrated management systems, campus card management and dining management, wherein the reliability of the historical data of the financial integrated management systems, the campus card management and the dining management is respectively 0.9, 0.92 and 0.95, and the campus card management and the dining management are used as reliable service systems;
Then, determining data sources corresponding to different service data based on the types of the service data of different service systems to be synchronously processed, wherein the number of the service data of campus card management and dining management is 9 and 13 respectively, so that the data reliability of different reliable service systems and the determination of a reference service system can be carried out through the number of the data sources in a reliable management system of campus card management and dining management, the associated other service systems of the service data corresponding to the different data sources and the historical data reliability, the data reliability of a specific campus card management system is 0.93, and the campus card management system is used as the reference service system;
Then the business system except the reference business system is used as a synchronous business system, namely a financial integrated management system and a catering management system, the extraction difficulty of business data of different synchronous business systems is determined according to the data architecture of different synchronous business systems, the data architecture of the financial integrated management system and the data architecture of the financial integrated management system are different, the storage protocol of the data storage of the financial integrated management system is different, the extraction difficulty of matching can be determined through the storage protocol, and the synchronous processing sequence of different synchronous business systems is determined by combining the historical data reliability of different synchronous business systems and the association relation of business data in the reference business system;
And finally, carrying out synchronous processing on the service data of the service systems based on the synchronous processing sequence until the synchronous processing of the service data of all the service systems is completed, and uniformly displaying the service data of different service systems based on the service requirements of users.
System class embodiment
In another aspect, as shown in FIG. 5, the present invention provides a computer system comprising: a communicatively coupled memory and processor, and a computer program stored on the memory and capable of running on the processor, characterized by: the processor executes a data synchronization management method as described above when running the computer program.
The data synchronization management method specifically comprises the following steps:
The method comprises the steps of determining the historical data reliability of different service systems and the reliable service systems by utilizing the data abnormality of different service data in different service systems during the historical synchronous processing;
Determining data sources corresponding to different service data based on the types of the service data of different service systems to be synchronously processed;
S21, judging whether the historical data reliability threshold of the reliable service system is smaller than a preset reliability threshold, if so, entering the next step, and if not, entering the step S24;
S22, judging whether the number of the data sources of the reliable service system is smaller than the number of the preset data sources, if so, determining that the reliable service system does not belong to a reference service system, and if not, entering step S23;
S23, determining the weight value of different data sources according to the number of other associated service systems of service data corresponding to the different data sources, dividing the data sources into associated data sources and general data sources based on the weight values of the data sources, determining the data source reliability of the associated data sources of the reliable service system according to the number of the associated data sources of the reliable service system and the weight values of the different associated data sources, judging whether the data source reliability of the associated data sources meets the requirement, if yes, determining that the service system does not belong to a reference service system, and if no, entering the next step;
S24, determining the data source reliability of the general data source of the reliable service system according to the number of the general data sources of the reliable service system and the weight values of different general data sources, determining the data reliability of the reliable service system according to the data source reliability of the general data source, the data source reliability of the related data source and the historical data reliability, and determining whether the reliable service system belongs to a reference service system according to the data reliability;
The service system except the reference service system is used as a synchronous service system, the extraction difficulty of service data of different synchronous service systems is determined according to the data architecture of different synchronous service systems, and the synchronous processing sequence of different synchronous service systems is determined by combining the correlation relation between the historical data reliability of different synchronous service systems and the service data in the reference service system;
And carrying out synchronous processing on the service data of the service systems based on the synchronous processing sequence until the synchronous processing of the service data of all the service systems is completed, and uniformly displaying the service data of different service systems based on the service requirements of users.
Through the above embodiments, the present invention has the following beneficial effects:
1. The method and the device utilize the abnormal data conditions of different service data in the service system in the history synchronous processing to determine the reliable service system, realize the screening of the reliable service system with less update failure times and less times of non-update success times, and lay a foundation for the screening of the reference service system with higher reliability.
2. The method comprises the steps of determining a reference service system according to the number of data sources, other service systems associated with service data corresponding to different data sources and the reliability of historical data, singly considering the reliability of the historical data of the reference service system, and screening the service systems with more data sources by further considering the distribution condition of the data sources in the service system, so that the reliability of the use of the reference service system is ensured.
3. Based on the extraction difficulty, the correlation relation between the historical data reliability and the service data in the reference service system, the synchronous processing sequence of different synchronous service systems is determined, the determination of the synchronous processing sequence of the synchronous service systems with the data reliability, the extraction difficulty and the correlation condition with the reference service system differentiated from three angles is realized, and the synchronous processing efficiency is ensured.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for apparatus, devices, non-volatile computer storage medium embodiments, the description is relatively simple, as it is substantially similar to method embodiments, with reference to the section of the method embodiments being relevant.
The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
The foregoing is merely one or more embodiments of the present description and is not intended to limit the present description. Various modifications and alterations to one or more embodiments of this description will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, or the like, which is within the spirit and principles of one or more embodiments of the present description, is intended to be included within the scope of the claims of the present description.
Claims (9)
1. The data synchronization management method is characterized by comprising the following steps of:
The method comprises the steps of determining the historical data reliability of different service systems and the reliable service systems by utilizing the data abnormality of different service data in different service systems during the historical synchronous processing;
determining data sources corresponding to different service data based on the types of the service data of different service systems to be synchronously processed, acquiring the number of the data sources in different reliable service systems, and determining the data reliability of different reliable service systems and the reference service system by combining the associated other service systems of the service data corresponding to the different data sources and the historical data reliability;
The service system except the reference service system is used as a synchronous service system, the extraction difficulty of service data of different synchronous service systems is determined according to the data architecture of different synchronous service systems, and the synchronous processing sequence of different synchronous service systems is determined by combining the correlation relation between the historical data reliability of different synchronous service systems and the service data in the reference service system;
Carrying out synchronous processing on the service data of the service systems based on the synchronous processing sequence until the synchronous processing of the service data of all the service systems is completed, and uniformly displaying the service data of different service systems based on the service demands of users;
the method for determining the reference service system comprises the following steps:
When the number of data sources in the reliable service system is smaller than the preset number of data sources and the historical data reliability threshold of the reliable service system is smaller than the preset reliability threshold: determining that the reliable service system does not belong to a reference service system;
determining weight values of different data sources according to the number of other associated service systems of service data corresponding to the different data sources, and dividing the data sources into associated data sources and general data sources based on the weight values of the data sources;
when the number of the data sources in the reliable service system is smaller than the preset number of the data sources or the historical data reliability threshold of the reliable service system is smaller than the preset reliability threshold:
Determining the reliability of the data sources of the associated data sources of the reliable service system according to the number of the associated data sources of the reliable service system and the weight values of different associated data sources, judging whether the reliability of the data sources of the associated data sources meets the requirement, if so, determining that the service system does not belong to a reference service system, and if not, entering the next step;
When the number of the data sources in the reliable service system is not less than the preset number of the data sources and the historical data reliability threshold of the reliable service system is not less than the preset reliability threshold:
And determining the data source reliability of the general data source of the reliable service system according to the number of the general data sources of the reliable service system and the weight values of different general data sources, determining the data reliability of the reliable service system based on the data source reliability of the general data source, the data source reliability of the related data source and the historical data reliability, and determining whether the reliable service system belongs to a reference service system according to the data reliability.
2. The data synchronization management method according to claim 1, wherein the data abnormality condition includes a data abnormality type and a number of data abnormality times of different data abnormality types.
3. The data synchronization management method of claim 2, wherein the data exception type comprises an unepdated, data update exception.
4. The data synchronization management method as claimed in claim 1, wherein the method for determining the reliable service system is:
Determining the historical data reliability of different business data of a business system based on the data anomaly types of different business data of the business system and the data anomaly times of different data anomaly types, and determining the system data reliability of the business system by combining the number of business systems associated with different business data;
Determining the synchronization processing reliability of the service system according to the quantity ratio of service data with data abnormality and the data abnormality type of the service system under different historical synchronization processing times, and determining the synchronization reliability of the service system by combining the moments corresponding to the different historical synchronization processing times;
And determining the historical data reliability of the reliable service system according to the system data reliability and the synchronous reliability of the service system, and determining whether the service system is the reliable service system or not according to the historical data reliability.
5. The data synchronization management method according to claim 1, wherein when the historical data reliability of the service system is greater than a preset reliability threshold, the service system is determined to be a reliable service system.
6. The method for synchronously managing data according to claim 1, wherein the method for determining the difficulty of extracting the service data of the synchronous service system comprises the steps of:
And determining a data protocol of the synchronous service system according to the data architecture of the synchronous service system, and determining the extraction difficulty of service data of the synchronous service system according to the data protocol.
7. The method for data synchronization management according to claim 1, wherein the method for determining the synchronization processing sequence of the synchronous service system comprises the steps of:
determining the association quantity of different synchronous service systems and service data in the reference service system according to the association relation between the different synchronous service systems and the service data in the reference service system, and determining the service association of the synchronous service systems and the reference service system by combining the quantity of data sources in the associated service data;
and determining a processing priority value of the synchronous service system based on the data extraction difficulty, the historical data reliability and the service relevance, and determining a synchronous processing sequence of the synchronous service system based on the processing priority value.
8. The method for data synchronization management according to claim 1, wherein the method for determining the synchronization processing sequence of the synchronous service system comprises the steps of:
determining the association quantity of different synchronous service systems and service data in the reference service system according to the association relation between the different synchronous service systems and the service data in the reference service system, and determining the service association of the synchronous service systems and the reference service system by combining the quantity of data sources in the associated service data;
And determining the processing priority of the synchronous service system through the historical data processing reliability of the synchronous service system, and determining the processing priority value of the synchronous service system based on the data extraction difficulty and the service relevance when the processing priority of the synchronous service system is the same, and determining the synchronous processing sequence of the synchronous service system based on the processing priority value.
9. A computer system, comprising: a communicatively coupled memory and processor, and a computer program stored on the memory and capable of running on the processor, characterized by: the processor, when running the computer program, performs a method of data synchronization management as claimed in any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410077035.9A CN117591607B (en) | 2024-01-19 | 2024-01-19 | Data synchronization management method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410077035.9A CN117591607B (en) | 2024-01-19 | 2024-01-19 | Data synchronization management method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117591607A CN117591607A (en) | 2024-02-23 |
| CN117591607B true CN117591607B (en) | 2024-05-07 |
Family
ID=89915453
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410077035.9A Active CN117591607B (en) | 2024-01-19 | 2024-01-19 | Data synchronization management method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117591607B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108808672A (en) * | 2018-07-10 | 2018-11-13 | 国网安徽省电力有限公司 | A kind of protective device reliability on-line prediction method and system based on guarantor's letter system |
| CN115328997A (en) * | 2022-07-15 | 2022-11-11 | 深圳市数帝网络科技有限公司 | Data synchronization method, system, device and storage medium |
| CN116166813A (en) * | 2022-12-15 | 2023-05-26 | 深圳银兴智能数据有限公司 | Management method, system, equipment and storage medium for big data automation operation and maintenance |
| CN116302487A (en) * | 2023-02-07 | 2023-06-23 | 南京博雅区块链研究院有限公司 | DOMA-based data center system and construction method |
| WO2023179042A1 (en) * | 2022-03-21 | 2023-09-28 | 中兴通讯股份有限公司 | Data updating method, fault diagnosis method, electronic device, and storage medium |
| CN116991947A (en) * | 2023-09-22 | 2023-11-03 | 湖南三湘银行股份有限公司 | Automatic data synchronization method and system |
| CN117149884A (en) * | 2023-10-30 | 2023-12-01 | 华兴国创(北京)科技有限公司 | Data processing transaction method |
| CN117294657A (en) * | 2023-11-24 | 2023-12-26 | 杭银消费金融股份有限公司 | Flow control method and device |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9411864B2 (en) * | 2008-08-26 | 2016-08-09 | Zeewise, Inc. | Systems and methods for collection and consolidation of heterogeneous remote business data using dynamic data handling |
| CN107729366B (en) * | 2017-09-08 | 2021-01-05 | 广东省建设信息中心 | Universal multi-source heterogeneous large-scale data synchronization system |
| US20230395253A1 (en) * | 2018-05-16 | 2023-12-07 | Shanghai Svm Medical Technology Co., Ltd. | Cloud-edge collaborative processing system and method for icu data based on third-party business |
-
2024
- 2024-01-19 CN CN202410077035.9A patent/CN117591607B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108808672A (en) * | 2018-07-10 | 2018-11-13 | 国网安徽省电力有限公司 | A kind of protective device reliability on-line prediction method and system based on guarantor's letter system |
| WO2023179042A1 (en) * | 2022-03-21 | 2023-09-28 | 中兴通讯股份有限公司 | Data updating method, fault diagnosis method, electronic device, and storage medium |
| CN115328997A (en) * | 2022-07-15 | 2022-11-11 | 深圳市数帝网络科技有限公司 | Data synchronization method, system, device and storage medium |
| CN116166813A (en) * | 2022-12-15 | 2023-05-26 | 深圳银兴智能数据有限公司 | Management method, system, equipment and storage medium for big data automation operation and maintenance |
| CN116302487A (en) * | 2023-02-07 | 2023-06-23 | 南京博雅区块链研究院有限公司 | DOMA-based data center system and construction method |
| CN116991947A (en) * | 2023-09-22 | 2023-11-03 | 湖南三湘银行股份有限公司 | Automatic data synchronization method and system |
| CN117149884A (en) * | 2023-10-30 | 2023-12-01 | 华兴国创(北京)科技有限公司 | Data processing transaction method |
| CN117294657A (en) * | 2023-11-24 | 2023-12-26 | 杭银消费金融股份有限公司 | Flow control method and device |
Non-Patent Citations (4)
| Title |
|---|
| Li Huagui ; Song Heliang et al.. Reliable Synchronization Technology for Continuous Variable Quantum Key Distribution System. 2022 International Conference on 6G Communications and IoT Technologies (6GIoTT).2023,第35-41页. * |
| 地理信息系统业务数据同步方法设计与实现;张书华;;油气田地面工程;20160720(07);第41-44页 * |
| 基于XML的分布式异构数据库同步系统设计与实现;方焕;;广西通信技术;20141215(04);第21-25页 * |
| 行业应用网关容灾系统建设研究;刘志，孙忠岩；;内蒙古通信;20160630(02);第37-42页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117591607A (en) | 2024-02-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107453960B (en) | Method, device and system for processing test data in service test | |
| CN111177222B (en) | Model testing method and device, computing equipment and storage medium | |
| CN110503551B (en) | Network fund transaction channel maintenance method, device and equipment | |
| US9430219B2 (en) | Revision safe upgrade in a hybrid cloud landscape | |
| CN108805667A (en) | Order flow processing method and system | |
| CN105787058B (en) | A kind of user tag system and the data delivery system based on user tag system | |
| CN103310460A (en) | Image characteristic extraction method and system | |
| CN105719112A (en) | Determination method and device for distribution abnormal state, and server | |
| CN113094434A (en) | Database synchronization method, system, device, electronic equipment and medium | |
| CN110334059A (en) | Method and apparatus for handling file | |
| CN110187908A (en) | A kind of method and system that GPU driver automatically updates | |
| CN109522043A (en) | A kind of management method of configuration data, device and storage medium | |
| CN117591607B (en) | Data synchronization management method and system | |
| CN111796984A (en) | Data monitoring method and device, computer equipment and storage medium | |
| CN109274991A (en) | The EMS memory management process and system of smart television | |
| CN114490137A (en) | Service data real-time statistical method and device, electronic equipment and readable storage medium | |
| CN114650211A (en) | Fault repairing method, device, electronic equipment and computer readable storage medium | |
| CN108121730A (en) | A kind of device and method by data update Fast synchronization to operation system | |
| CN112035366A (en) | Test case generation method, device and equipment | |
| CN113609236B (en) | Data processing method, device and equipment | |
| CN109376141A (en) | A kind of data migration method and device | |
| CN111930620B (en) | Application running environment data processing method and device | |
| CN114219608A (en) | Post-loan anti-fraud processing method and device and computer equipment | |
| CN114201501A (en) | Message processing method and device, electronic equipment and storage medium | |
| CN118041822A (en) | View generation method, program product and storage medium for research and development test environment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: Room 301, 3rd Floor, Building 6, West Lake International Science and Technology Building, No. 391 Wen'er Road, Cuiyuan Street, Xihu District, Hangzhou City, Zhejiang Province, 310000 Patentee after: HANGZHOU GREEN OLIVES NETWORK TECHNOLOGY CO.,LTD. Country or region after: China Address before: 310000 room 316, floor 3, building 6, Xihu international science and technology building, No. 391, Wener Road, Xihu District, Hangzhou, Zhejiang Province Patentee before: HANGZHOU GREEN OLIVES NETWORK TECHNOLOGY CO.,LTD. Country or region before: China |