CN107689998B - Incremental data synchronization method, terminal equipment and computer readable storage medium - Google Patents
Incremental data synchronization method, terminal equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN107689998B CN107689998B CN201710825934.2A CN201710825934A CN107689998B CN 107689998 B CN107689998 B CN 107689998B CN 201710825934 A CN201710825934 A CN 201710825934A CN 107689998 B CN107689998 B CN 107689998B
- Authority
- CN
- China
- Prior art keywords
- incremental
- data
- incremental data
- record
- service subsystem
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Abstract
The invention belongs to the technical field of computers, and particularly relates to an incremental data synchronization method and terminal equipment. The method comprises the following steps: respectively acquiring first incremental data from each service subsystem, wherein the first incremental data are user data actually added to each service subsystem in a preset first time period; summarizing the acquired first incremental data to obtain second incremental data, wherein the second incremental data is user data which is newly added to each service subsystem in the first time period; comparing each first incremental data with each second incremental data to obtain an incremental record missing from each service subsystem, wherein the incremental record is a basic unit of the incremental data; and respectively adding the missing incremental records to the corresponding service subsystems to complete the synchronization of the incremental data of each service subsystem. By the method and the system, the consistency of the incremental data among the service subsystems is ensured.
Description
Technical Field
The invention belongs to the technical field of computers, and particularly relates to an incremental data synchronization method and terminal equipment.
Background
The existing financial service subsystems are large in scale and complex in structure, the whole system is often divided into a plurality of service subsystems according to service classification, therefore, incremental data of users often need to be synchronized in the plurality of service subsystems, the existing mode mainly records the incremental data to a plurality of different service subsystems in real time through an interface calling mode, the consistency of data synchronization is difficult to guarantee, and the error rate is high.
Disclosure of Invention
In view of this, embodiments of the present invention provide an incremental data synchronization method and a terminal device, so as to solve the problems that consistency of data synchronization is difficult to guarantee and an error rate is high when incremental data is recorded to multiple different service subsystems in real time through an interface call mode.
A first aspect of an embodiment of the present invention provides an incremental data synchronization method, which may include:
respectively acquiring first incremental data from each service subsystem, wherein the first incremental data are user data actually added to each service subsystem in a preset first time period;
summarizing the acquired first incremental data to obtain second incremental data, wherein the second incremental data is user data which is newly added to each service subsystem in the first time period;
comparing each first incremental data with each second incremental data to obtain the missing incremental record of each service subsystem, wherein the incremental record is a basic unit of the incremental data, and each incremental record comprises a piece of service information;
and respectively adding the missing incremental records to the corresponding service subsystems to complete the synchronization of the incremental data of each service subsystem.
A second aspect of embodiments of the present invention provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of:
respectively acquiring first incremental data from each service subsystem, wherein the first incremental data are user data actually added to each service subsystem in a preset first time period;
summarizing the acquired first incremental data to obtain second incremental data, wherein the second incremental data is user data which is newly added to each service subsystem in the first time period;
comparing each first incremental data with each second incremental data to obtain the missing incremental record of each service subsystem, wherein the incremental record is a basic unit of the incremental data, and each incremental record comprises a piece of service information;
and respectively adding the missing incremental records to the corresponding service subsystems to complete the synchronization of the incremental data of each service subsystem.
A third aspect of the embodiments of the present invention provides an incremental data synchronization terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the following steps when executing the computer program:
respectively acquiring first incremental data from each service subsystem, wherein the first incremental data are user data actually added to each service subsystem in a preset first time period;
summarizing the acquired first incremental data to obtain second incremental data, wherein the second incremental data is user data which is newly added to each service subsystem in the first time period;
comparing each first incremental data with each second incremental data to obtain the missing incremental record of each service subsystem, wherein the incremental record is a basic unit of the incremental data, and each incremental record comprises a piece of service information;
and respectively adding the missing incremental records to the corresponding service subsystems to complete the synchronization of the incremental data of each service subsystem.
Compared with the prior art, the embodiment of the invention has the following beneficial effects: according to the embodiment of the invention, the incremental data are firstly acquired from each service subsystem respectively and then are summarized, the summarized incremental data provide a synchronous standard, the incremental data of each service subsystem are compared with the incremental data respectively to obtain the difference records between the incremental data and the synchronous standard, and because the differences are obtained based on the same synchronous standard, the incremental data among the service subsystems can be ensured to be consistent as long as the service subsystems synchronously process the respective incremental data, and because the synchronous standard summarizes the data of each service subsystem, the accuracy of the data is greatly ensured, and the error rate during synchronization can be greatly reduced by taking the data as reference.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a schematic diagram of an implementation environment in an embodiment of the invention;
FIG. 2 is a flowchart of an embodiment of a method for incremental data synchronization according to an embodiment of the present invention;
FIG. 3 is a flowchart of an embodiment of an incremental data synchronization method in a scenario considering data errors according to an embodiment of the present invention;
FIG. 4 is a schematic flow chart of determining a system to be investigated in an embodiment of the present invention;
fig. 5 is a schematic block diagram of an incremental data synchronization terminal device according to an embodiment of the present invention;
fig. 6 is a program block diagram of an incremental data synchronization procedure according to an embodiment of the present invention.
Detailed Description
In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the embodiments described below are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a schematic diagram of an implementation environment according to an embodiment of the present invention, where an execution subject in the embodiment of the present invention is a synchronous processing system shown in the diagram, the synchronous processing system is connected to each service subsystem, reads and summarizes incremental data of each service subsystem, compares the incremental data of each service subsystem with the summarized incremental data to obtain missing incremental data of each service subsystem, and finally adds the missing incremental records to the corresponding service subsystems, so as to complete synchronization of the incremental data of each service subsystem.
Referring to fig. 2, an embodiment of an incremental data synchronization method according to an embodiment of the present invention may include:
step S201, obtaining first incremental data from each service subsystem respectively.
In this embodiment, the synchronous processing system may read the first incremental data of each service subsystem from a preset first storage location at a preset first time, and the first incremental data is extracted and stored in the first storage location by each service subsystem at a preset second time.
For example, a company routinely synchronizes incremental data once every morning, each business subsystem starts a timing task at T +1 morning 00:05:00 (i.e., the second time), extracts incremental data (i.e., the first incremental data) of T days 00:00:00 to 24:00:00 (i.e., the first time period) in each system, and stores the extracted incremental data in the first storage location. The synchronous processing system starts a timing task at the morning time of T +1 (namely the first time) of 02:00:00, and the first incremental data is read from the first storage position. In order to ensure that the synchronous processing system can smoothly read data and avoid the situation that no data is stored in the first storage position during reading, the first time is later than the second time, the interval between the first time and the second time is larger than a preset time interval, and the specific value of the time interval can be set according to the actual situation so as to ensure that the data extraction, storage and other processes of each service subsystem are enough.
Step S202, summarizing the acquired first incremental data to obtain second incremental data.
And the second incremental data is user data which is newly added by each service subsystem in the first time period.
In this embodiment, the incremental data is composed of a plurality of incremental records, where the incremental records are basic units of the incremental data, and each incremental record contains a piece of service information, for example, each incremental record may represent a binding relationship between a user and a product account.
When summarizing the read first incremental data, each incremental data can be regarded as a set, increments are recorded as elements in the set, and then a union of the sets, that is, the second incremental data is obtained. Specifically, all incremental records may be extracted and combined into one incremental data, and then only one of the incremental records is reserved for the plurality of incremental records containing the same service information, and other duplicate incremental records are deleted.
For example, if the first incremental data of the service subsystem 1 includes: incremental record 1, incremental record 2, incremental record 3, and incremental record 4, the first incremental data of the business subsystem 2 comprising: the incremental record 1, the incremental record 4, the incremental record 5, and the incremental record 6, and then the second incremental data obtained after the aggregation includes: incremental record 1, incremental record 2, incremental record 3, incremental record 4, incremental record 5, and incremental record 6.
Step S203, comparing each of the first incremental data with the second incremental data, to obtain the missing incremental record of each of the service subsystems.
In the above example, the first incremental data of the service subsystem 1 is compared with the second incremental data, the obtained missing incremental records are incremental record 5 and incremental record 6, and the first incremental data of the service subsystem 2 is compared with the second incremental data, and the obtained missing incremental records are incremental record 2 and incremental record 3.
And step S204, adding the missing incremental records to the corresponding service subsystems respectively to complete the synchronization of the incremental data of each service subsystem.
Specifically, the synchronous processing system may store the missing incremental records into a preset second storage location at a preset third time, so that each service subsystem reads the respective missing incremental record from the second storage location at a preset fourth time, and adds the missing incremental record to the corresponding first incremental data to obtain the synchronized incremental data.
It should be understood that the third time should be later than the first time, and the interval between the first time and the third time should be greater than a certain time interval, so as to ensure that enough synchronous processing systems complete the reading, summarizing, comparing and other processes.
For example, the synchronous processing system starts the processes of reading, summarizing, comparing and the like of the timing task at the early morning of the T +1 day 02:00:00 (namely, the first time), starts the processes of storing the timing task at the early morning of the T +1 day 05:00:00 (namely, the third time), and starts the processes of reading and adding the timing task at the early morning of the T +1 day 06:00:00 (namely, the fourth time).
In the above embodiments, it is based on the premise that the data of the service subsystem is trusted data. In practical applications, however, data of the service subsystem may be in error, which needs to be considered. Referring to FIG. 3, one embodiment under consideration of a scenario where there is a data error may include:
step S301, obtaining first incremental data from each service subsystem respectively.
Step S301 is the same as step S201 in the foregoing embodiment, and specific reference may be made to the description in the foregoing embodiment, which is not repeated herein.
Step S302, summarizing the acquired first incremental data to obtain second incremental data.
Step S302 is the same as step S202 in the above embodiment, and reference may be specifically made to the description in the above embodiment, which is not repeated herein.
Step S303, calculating the reliability of each incremental record in the second incremental data according to the preset data reliability of each service subsystem.
The data reliability of each service subsystem is determined according to historical statistical data, and if the more operations (including an operation of adding an increment record and an operation of deleting the increment record) are performed on the increment data by the service subsystem in the historical statistical data, the higher the probability of error of the data is, and the lower the data reliability is. The value range of the data reliability of the service subsystem is [0, 1], when 0 is taken, the data is completely untrustworthy, and when 1 is taken, the data is completely credible.
In this embodiment, the reliability of the incremental record may be calculated as follows: and summing the data credibility of each service subsystem of the incremental record, dividing the sum by the total number of the service subsystems, and obtaining a final quotient which is the credibility of the incremental record, wherein similarly, the value range of the credibility of the incremental record is [0, 1], when 0 is taken, the data is completely untrustworthy, and when 1 is taken, the data is completely credible.
For example, if there are 3 service subsystems, namely, service subsystem 1, service subsystem 2, and service subsystem 3, the data reliability is 0.9, 0.8, and 0.7. If incremental record 1 is present in service subsystem 2 and service subsystem 3, but not in service subsystem 1, the reliability of incremental record 1 is (0.8+ 0.7)/3-0.5.
Step S304, extracting a preferred record with the reliability greater than a preset second threshold from each incremental record in the second incremental data.
The value of the second threshold may be set according to an actual situation, which is not specifically limited in this embodiment, but it should be noted that if the value is larger, the extracted preferred records are fewer, and if the value is smaller, the extracted preferred records are more.
Alternatively, the following processing manner may be adopted as a simplification of step S303 and step S304: respectively counting the total times of the increment records in the second increment data appearing in the first increment data; and extracting the preferred records of which the total times are greater than a preset first threshold value from each incremental record in the second incremental data, wherein the first threshold value is determined by the total number of the service subsystems. For example, if the total number of the service subsystems is 10, the first threshold is set to be half of the total number, that is, the value is 5, the total number of times that the incremental record 1 appears in the first incremental data is 4, and the total number of times that the incremental record 2 appears in the first incremental data is 6, then the incremental record 2 is the preferred record, and the incremental record 1 is not the preferred record.
And step S305, combining the extracted preferred records into preferred second incremental data.
Compared with the initial second incremental data, the preferable second incremental data only keep the incremental records with higher credibility, and accordingly, the obtained synchronization result is more accurate.
Step S306, comparing each of the first incremental data with the optimized second incremental data, to obtain missing incremental records and redundant incremental records of each of the service subsystems.
For example, if the first incremental data of the service subsystem 1 includes: incremental record 1, incremental record 2, incremental record 3, and incremental record 4, the first incremental data of the business subsystem 2 comprising: increment record 1, increment record 2, increment record 3 and increment record 5, and the preferred second increment data includes: incremental record 1, incremental record 2, incremental record 3, and incremental record 5.
Comparing the first incremental data of the service subsystem 1 with the optimized second incremental data to obtain a missing incremental record 5 and a redundant incremental record 4, and comparing the first incremental data of the service subsystem 2 with the second incremental data, wherein the two are completely consistent, and no missing incremental record or redundant incremental record exists.
Step S307, the missing incremental records are respectively added to the corresponding service subsystems, and the redundant incremental records are respectively deleted from the corresponding service subsystems.
Specifically, the synchronous processing system may store the missing incremental record and the redundant incremental record into preset second storage locations, so that each service subsystem reads the missing incremental record and the redundant incremental record from the second storage location, adds the missing incremental record to the corresponding first incremental data, and deletes the redundant incremental record from the corresponding first incremental data, thereby obtaining the synchronized incremental data.
Preferably, for the service subsystem with low data credibility, the service subsystem needs to be checked to reduce the probability of error. Specifically, the method may include the steps shown in fig. 4:
step S401, respectively counting the total number of the historical operation records of each service subsystem.
The historical operation record is the operation record (including the operation of adding incremental record and the operation of deleting incremental record) executed by the service subsystem during the synchronization process within a preset second time period, and in order to avoid the influence of short-term burst conditions, the second time period should be long enough, for example, it may take one week, one month, and the like.
Step S402, determining the service subsystem with the total number of the historical operation records larger than a preset third threshold value as a system to be checked;
the larger the total number of the historical operation records is, the higher the probability of the data error is. The value of the third threshold may be set according to an actual situation, which is not specifically limited in this embodiment, but it should be noted that if the value is larger, the determined systems to be checked are fewer, and if the value is smaller, the determined systems to be checked are more.
And step S403, notifying the appointed staff to check the system to be checked.
In summary, according to the embodiments of the present invention, the incremental data are obtained from each service subsystem, and then summarized, the summarized incremental data provide a synchronization standard, the incremental data of each service subsystem are compared with each other, so as to obtain the difference records between the incremental data and the synchronization standard, because the differences are obtained based on the same synchronization standard, the incremental data between the service subsystems are guaranteed to be consistent as long as the service subsystems perform synchronization processing on the respective incremental data, and because the synchronization standard summarizes the data of each service subsystem, the accuracy is greatly guaranteed, and the error rate during synchronization can be greatly reduced by using the data as a reference.
Corresponding to the incremental data synchronization method described in the foregoing embodiment, fig. 5 shows a schematic block diagram of an incremental data synchronization terminal device provided in the embodiment of the present invention, and for convenience of description, only the parts related to the embodiment of the present invention are shown.
In this embodiment, the incremental data synchronization terminal device may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The incremental data synchronization terminal device may include: a processor 50, a memory 51 and a computer program 52 stored in said memory 51 and executable on said processor 50.
The Processor 50 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 51 may be an internal storage unit of the incremental data synchronization terminal device 5, such as a hard disk or a memory of the incremental data synchronization terminal device 5. The memory 51 may also be an external storage device of the incremental data synchronization terminal device 5, such as a plug-in hard disk, a Smart Memory Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are equipped on the incremental data synchronization terminal device 5. Further, the memory 51 may also include both an internal storage unit and an external storage device of the incremental data synchronization terminal device 5. The memory 51 is used for storing the computer program and other programs and data required by the incremental data synchronization terminal device 5. The memory 51 may also be used to temporarily store data that has been output or is to be output.
Please refer to fig. 6, which is a block diagram of a computer program 52 according to an embodiment of the present invention. In this embodiment, the computer program 52 may be divided into one or more modules, and the one or more modules are stored in the memory 51 and executed by the processor 50 to implement the present invention. For example, in fig. 6, the computer program 52, that is, the incremental data synchronization program, may be divided into an incremental data acquisition module 601, an incremental data summarization module 602, a first comparison module 603, and a missing record addition module 604. The following description will specifically describe the functions of the program modules 601-604.
An incremental data obtaining module 601, configured to obtain first incremental data from each service subsystem, where the first incremental data is user data actually added to each service subsystem within a preset first time period;
an incremental data summarizing module 602, configured to summarize each acquired first incremental data to obtain second incremental data, where the second incremental data is user data that should be newly added to each service subsystem in the first time period;
a first comparison module 603, configured to compare each of the first incremental data with the second incremental data, respectively, to obtain an incremental record missing from each of the service subsystems, where the incremental record is a basic unit of the incremental data, and each incremental record includes one piece of service information;
a missing record adding module 604, configured to add the missing incremental records to corresponding service subsystems, respectively, so as to complete synchronization of the incremental data of each service subsystem.
Optionally, the computer program 52 may further include:
the times counting module is used for respectively counting the total times of all incremental records in the second incremental data appearing in the first incremental data;
a first preferred record extracting module, configured to extract preferred records of which the total number of times is greater than a preset first threshold from each incremental record in the second incremental data, where the first threshold is determined by the total number of the service subsystems;
and the first composition module is used for composing the extracted preferable records into preferable second incremental data.
Optionally, the computer program 52 may further include:
the reliability calculation module is used for calculating the reliability of each incremental record in the second incremental data according to the preset data reliability of each service subsystem;
a second preferred record extracting module, configured to extract, from each incremental record in the second incremental data, a preferred record whose reliability is greater than a preset second threshold;
and the second composition module is used for composing the extracted preferable records into preferable second incremental data.
Further, the computer program 52 may further include:
the second comparison module is used for comparing each first incremental data with the optimized second incremental data respectively to obtain redundant incremental records of each service subsystem;
and the redundant record deleting module is used for deleting the redundant incremental records from the corresponding service subsystems respectively.
Further, the computer program 52 may further include:
the operation counting module is used for respectively counting the total number of historical operation records of each service subsystem, wherein the historical operation records are operation records executed when the service subsystems perform synchronous processing in a preset second time period;
the system to be checked determining module is used for determining the service subsystem of which the total number of the historical operation records is greater than a preset third threshold value as the system to be checked;
and the troubleshooting informing module is used for informing the appointed staff to perform troubleshooting on the system to be debugged.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art would appreciate that the modules, elements, and/or method steps of the various embodiments described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (6)
1. An incremental data synchronization method, comprising:
respectively acquiring first incremental data from each service subsystem, wherein the first incremental data are user data actually added to each service subsystem in a preset first time period;
summarizing the acquired first incremental data to obtain second incremental data, wherein the second incremental data is user data which should be newly added to each service subsystem in the first time period, and the second incremental data is a union of the first incremental data;
calculating the reliability of each incremental record in the second incremental data according to the preset data reliability of each service subsystem, specifically, calculating the reliability of the incremental record by the following method: summing the data reliability of each service subsystem of the incremental record, dividing the sum by the total number of the service subsystems, wherein the final quotient is the reliability of the incremental record, the data reliability of each service subsystem is determined according to historical statistical data, and if the historical statistical data contains more operations for modifying the incremental data by the service subsystems, the data reliability is lower;
extracting a preferred record with the reliability being greater than a preset second threshold value from each incremental record in the second incremental data;
composing the extracted preferred records into preferred second incremental data;
comparing each first incremental data with the optimized second incremental data to obtain missing incremental records of each service subsystem, wherein the incremental records are basic units of the incremental data, and each incremental record comprises a piece of service information;
and respectively adding the missing incremental records to the corresponding service subsystems to complete the synchronization of the incremental data of each service subsystem.
2. The incremental data synchronization method of claim 1, further comprising:
comparing each first incremental data with the optimized second incremental data to obtain redundant incremental records of each service subsystem;
and deleting the redundant incremental records from the corresponding service subsystems respectively.
3. The incremental data synchronization method of claim 1, further comprising:
respectively counting the total number of historical operation records of each service subsystem, wherein the historical operation records are operation records executed when the service subsystems perform synchronous processing in a preset second time period;
determining the service subsystem with the total number of the historical operation records larger than a preset third threshold value as a system to be checked;
and informing the appointed staff to check the system to be checked.
4. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the incremental data synchronization method according to any one of claims 1 to 3.
5. An incremental data synchronization terminal device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, characterized in that said processor implements the following steps when executing said computer program:
respectively acquiring first incremental data from each service subsystem, wherein the first incremental data are user data actually added to each service subsystem in a preset first time period;
summarizing the acquired first incremental data to obtain second incremental data, wherein the second incremental data is user data which should be newly added to each service subsystem in the first time period, and the second incremental data is a union of the first incremental data;
calculating the reliability of each incremental record in the second incremental data according to the preset data reliability of each service subsystem, specifically, calculating the reliability of the incremental record by the following method: summing the data reliability of each service subsystem of the incremental record, dividing the sum by the total number of the service subsystems, wherein the final quotient is the reliability of the incremental record, the data reliability of each service subsystem is determined according to historical statistical data, and if the historical statistical data contains more operations for modifying the incremental data by the service subsystems, the data reliability is lower;
extracting a preferred record with the reliability being greater than a preset second threshold value from each incremental record in the second incremental data;
composing the extracted preferred records into preferred second incremental data;
comparing each first incremental data with the optimized second incremental data to obtain missing incremental records of each service subsystem, wherein the incremental records are basic units of the incremental data, and each incremental record comprises a piece of service information;
and respectively adding the missing incremental records to the corresponding service subsystems to complete the synchronization of the incremental data of each service subsystem.
6. The incremental data synchronization terminal device of claim 5, further comprising:
respectively counting the total number of historical operation records of each service subsystem, wherein the historical operation records are operation records executed when the service subsystems perform synchronous processing in a preset second time period;
determining the service subsystem with the total number of the historical operation records larger than a preset third threshold value as a system to be checked;
and informing the appointed staff to check the system to be checked.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710825934.2A CN107689998B (en) | 2017-09-14 | 2017-09-14 | Incremental data synchronization method, terminal equipment and computer readable storage medium |
| PCT/CN2018/074869 WO2019052112A1 (en) | 2017-09-14 | 2018-01-31 | Incremental data synchronization method, readable storage medium, terminal device, and apparatus |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710825934.2A CN107689998B (en) | 2017-09-14 | 2017-09-14 | Incremental data synchronization method, terminal equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107689998A CN107689998A (en) | 2018-02-13 |
| CN107689998B true CN107689998B (en) | 2020-02-14 |
Family
ID=61156265
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710825934.2A Active CN107689998B (en) | 2017-09-14 | 2017-09-14 | Incremental data synchronization method, terminal equipment and computer readable storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN107689998B (en) |
| WO (1) | WO2019052112A1 (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108573042B (en) * | 2018-04-10 | 2022-06-10 | 平安科技（深圳）有限公司 | Report synchronization method, electronic equipment and computer readable storage medium |
| WO2020097835A1 (en) * | 2018-11-14 | 2020-05-22 | 华为技术有限公司 | Multimedia service transmission method and device |
| CN110781197B (en) * | 2019-09-29 | 2023-09-26 | 北京淇瑀信息科技有限公司 | Hive offline synchronous verification method and device and electronic equipment |
| CN110837537A (en) * | 2019-10-12 | 2020-02-25 | 平安银行股份有限公司 | Order data synchronization method and device, computer readable storage medium and server |
| CN110716939B (en) * | 2019-10-16 | 2023-05-09 | 深圳市网心科技有限公司 | Data management method, electronic device, system and medium |
| CN114116842B (en) * | 2021-11-25 | 2023-05-19 | 上海柯林布瑞信息技术有限公司 | Multidimensional medical data real-time acquisition method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101577817A (en) * | 2009-06-10 | 2009-11-11 | 中兴通讯股份有限公司 | Distributed video monitoring system and method thereof for keeping data consistency |
| CN102096685A (en) * | 2009-12-11 | 2011-06-15 | 阿里巴巴集团控股有限公司 | Method and device for synchronizing distributive data into data warehouse |
| CN103823884A (en) * | 2014-03-06 | 2014-05-28 | 国家电网公司 | Maintenance right-based loose consistency synchronization method for grid calculating data |
-
2017
- 2017-09-14 CN CN201710825934.2A patent/CN107689998B/en active Active
-
2018
- 2018-01-31 WO PCT/CN2018/074869 patent/WO2019052112A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101577817A (en) * | 2009-06-10 | 2009-11-11 | 中兴通讯股份有限公司 | Distributed video monitoring system and method thereof for keeping data consistency |
| CN102096685A (en) * | 2009-12-11 | 2011-06-15 | 阿里巴巴集团控股有限公司 | Method and device for synchronizing distributive data into data warehouse |
| CN103823884A (en) * | 2014-03-06 | 2014-05-28 | 国家电网公司 | Maintenance right-based loose consistency synchronization method for grid calculating data |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019052112A1 (en) | 2019-03-21 |
| CN107689998A (en) | 2018-02-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107689998B (en) | Incremental data synchronization method, terminal equipment and computer readable storage medium | |
| CN113326165B (en) | Data processing method and device based on block chain and computer readable storage medium | |
| CN108197002B (en) | Mobile device non-buried point data statistical method, system, terminal and medium | |
| CN111400170A (en) | Data permission testing method and device | |
| CN109284331B (en) | Certificate making information acquisition method based on service data resources, terminal equipment and medium | |
| CN106358220B (en) | The detection method of abnormal contact information, apparatus and system | |
| CN113220777B (en) | Service data processing method, device, computer equipment and storage medium | |
| CN111625532A (en) | Data blood relationship processing method and device, computer equipment and storage medium | |
| CN110046172B (en) | Online computing data processing method and system | |
| CN116775575A (en) | File merging method and device, electronic equipment and storage medium | |
| CN109324963B (en) | Method for automatically testing profit result and terminal equipment | |
| CN107844273B (en) | Data writing method and device and verification method and device | |
| WO2019062087A1 (en) | Attendance check data testing method, terminal and device, and computer readable storage medium | |
| CN112367540B (en) | Method and device for monitoring live broadcast stream pushing number on line | |
| CN112163127B (en) | Relationship graph construction method and device, electronic equipment and storage medium | |
| CN115129809A (en) | Method and device for determining user activity, electronic equipment and storage medium | |
| CN114238335A (en) | Buried point data generation method and related equipment thereof | |
| CN112835854A (en) | File storage method and device, electronic equipment and storage medium | |
| CN110703988B (en) | Storage pool creating method, system, terminal and storage medium for distributed storage | |
| CN109522211B (en) | Interface parameter transmission method and device, electronic equipment and storage medium | |
| CN110837537A (en) | Order data synchronization method and device, computer readable storage medium and server | |
| CN111626868A (en) | Account checking method, account checking device, account checking equipment and computer readable storage medium | |
| CN112488857A (en) | Event recognition method and device, electronic equipment and storage medium | |
| CN111381977A (en) | Message processing method and device | |
| CN109447386B (en) | Work assessment method and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |