CN115643271A - Method, device, server and medium for synchronizing multi-application data on cloud - Google Patents
Method, device, server and medium for synchronizing multi-application data on cloud Download PDFInfo
- Publication number
- CN115643271A CN115643271A CN202211345921.2A CN202211345921A CN115643271A CN 115643271 A CN115643271 A CN 115643271A CN 202211345921 A CN202211345921 A CN 202211345921A CN 115643271 A CN115643271 A CN 115643271A
- Authority
- CN
- China
- Prior art keywords
- data
- data table
- synchronization
- abstract value
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method, a device, a server and a medium for synchronizing multi-application data on the cloud. And the data synchronization receiver receives the data synchronization request information, and calculates a second data abstract value of the received data table after acquiring the first data abstract value and the data table carried in the data synchronization request information. When the first data abstract value is judged to be consistent with the second data abstract value, the data table sent by the data synchronization initiator is completely consistent with the data table received by the data synchronization receiver, and then the data table is imported, so that the consistency of the data table synchronization is strictly ensured.
Description
Technical Field
The present application relates to the field of data transmission technologies, and in particular, to a method, an apparatus, a server, and a medium for synchronizing multiple application data in a cloud.
Background
The cloud native technology architecture solves the problem that the traditional application data synchronization needs special research and development in the aspects of application tight coupling, elastic expansion and contraction, observable capability and the like, and the cloud application data synchronization makes higher requirements on consistency and safety due to difficulty in troubleshooting caused by software platform hosting.
The prior art data synchronization technology mainly comprises application of tightly coupled data synchronization, timing data synchronization, message-middleware asynchronous data synchronization and database real-time data synchronization. And carrying data synchronization logic in the business logic by applying tightly coupled data synchronization to realize data synchronization. Timing data synchronization a timer is arranged outside the application service function, and data synchronization is realized through the timer. Asynchronous data synchronization of the message middleware class realizes data synchronization by using an asynchronous processing mode. The real-time data synchronization of the database realizes the data synchronization of the fields by means of a database rolling transaction log.
The four types of data synchronization modes are suitable for different types of service systems, but the data consistency is insufficient to different degrees, and the problem of inconsistent synchronization data may occur, so that the subsequent related services are influenced.
Disclosure of Invention
The embodiment of the application provides a method, a device, a server and a medium for synchronizing multiple application data on a cloud, and is used for solving the problem that data consistency of data synchronization is poor in the prior art.
The application provides a method for synchronizing multiple application data on a cloud, which comprises the following steps:
a data synchronization initiator acquires a data table to be synchronized, determines a first data abstract value of the data table through a Hash algorithm, stores the first data abstract value and the data table in data synchronization request information, and sends the data synchronization request information to a data synchronization receiver;
and the data synchronization receiver receives the data synchronization request information, acquires the first data abstract value and the data table carried in the data synchronization request information, determines a second data abstract value of the received data table through a Hash algorithm, judges whether the first data abstract value is consistent with the second data abstract value, and if so, performs import operation on the data table.
Further, the data synchronization initiator determines a data table to be synchronized, determines a first data digest value of the data table through a hash algorithm, and stores the first data digest value and the data table in data synchronization request information, including:
judging whether the current data synchronization service scene is a dependent service scene, if so, acquiring a dependent data table corresponding to the data table to be synchronized, determining a first data abstract value of the data table and a third data abstract value of the dependent data table through a hash algorithm, and storing the first data abstract value, the third data abstract value and the data table in data synchronization request information.
Further, the data synchronization receiving party receives the synchronization request information, acquires the first data digest value and the data table carried in the synchronization request information, determines a second data digest value of the received data table through a hash algorithm, and determines whether the first data digest value and the second data digest value are consistent, and if so, the importing operation on the data table includes:
the data synchronization receiver receives the synchronization request information and acquires the first data abstract value, the third data abstract value and the data table carried in the synchronization request information;
determining an imported dependent data table corresponding to the data table according to a preset import condition;
determining a second data digest value of the received data table and a fourth data digest value of the imported dependent data table through a hash algorithm;
and judging whether the first data abstract value is consistent with the second data abstract value, and whether the third data abstract value is consistent with the fourth data abstract value, and if so, importing the data table.
Further, before the importing operation is performed on the data table, the method further includes:
judging whether the imported data has the covered data corresponding to the data table or not, if so, backing up the covered data, storing the backed-up data to a specified address, deleting the covered data in the imported data, and importing the data table.
Further, the method further comprises:
and after the data table is imported, feeding back prompt information of synchronized data to the data synchronization initiator.
Further, the method further comprises:
and if the first data abstract value is judged to be inconsistent with the second data abstract value, returning the data table to the data synchronization initiator, and sending prompt information of abnormal synchronization data to a manager of the data synchronization initiator.
Further, the method further comprises:
and if the third data abstract value is judged to be inconsistent with the fourth data abstract value, returning the data table to the data synchronization initiator, and sending prompt information of the dependent abnormal synchronization task to a manager of the data synchronization initiator.
In another aspect, the present application provides a device for synchronizing multiple application data on a cloud, where the device includes:
the sending module is used for acquiring a data table to be synchronized, determining a first data abstract value of the data table through a Hash algorithm, storing the first data abstract value and the data table in data synchronization request information, and sending the data synchronization request information to a data synchronization receiver;
and the receiving module is used for receiving the data synchronization request information, acquiring the first data abstract value and the data table carried in the data synchronization request information, determining a second data abstract value of the received data table through a Hash algorithm, judging whether the first data abstract value is consistent with the second data abstract value, and if so, importing the data table.
Further, the sending module is specifically configured to determine whether a current data synchronization service scenario is a dependent service scenario, and if yes, obtain a dependent data table corresponding to the data table to be synchronized, determine a first data digest value of the data table and a third data digest value of the dependent data table through a hash algorithm, and store the first data digest value, the third data digest value, and the data table in data synchronization request information.
Further, the receiving module is specifically configured to receive the synchronization request information by the data synchronization receiving party, and obtain the first data digest value, the third data digest value, and the data table carried in the synchronization request information; determining an imported dependent data table corresponding to the data table according to a preset import condition; determining a second data digest value of the received data table and a fourth data digest value of the imported dependent data table through a hash algorithm; and judging whether the first data abstract value is consistent with the second data abstract value, and whether the third data abstract value is consistent with the fourth data abstract value, and if so, importing the data table.
Further, the receiving module is further configured to determine whether there is covered data corresponding to the data table in the imported data, if so, backup the covered data, store the backed-up data to a specified address, delete the covered data in the imported data, and perform an import operation on the data table.
Further, the apparatus further comprises:
and the first feedback module is used for feeding back prompt information of synchronized data to the data synchronization initiator after the data table is imported.
Further, the apparatus further comprises:
and the second feedback module is used for returning the data table to the data synchronization initiator and sending prompt information of abnormal synchronous data to a manager of the data synchronization initiator if the first data abstract value is judged to be inconsistent with the second data abstract value.
Further, the apparatus further comprises:
and the third feedback module is used for returning the data table to the data synchronization initiator and sending prompt information of the dependent abnormal synchronization task to a manager of the data synchronization initiator if the third data abstract value is judged to be inconsistent with the fourth data abstract value.
In another aspect, the present application provides a server, including a processor, a communication interface, a memory and a communication bus, where the processor, the communication interface and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of the above when executing a program stored in the memory.
In yet another aspect, the present application provides a computer-readable storage medium having a computer program stored therein, which when executed by a processor, performs the method steps of any one of the above.
The application provides a method, a device, a server and a medium for synchronizing multi-application data on a cloud, wherein the method comprises the following steps: a data synchronization initiator determines a data table to be synchronized, a first data abstract value of the data table is determined through a Hash algorithm, the first data abstract value and the data table are stored in data synchronization request information, and the data synchronization request information is sent to a data synchronization receiver; and the data synchronization receiver receives the synchronization request information, acquires the first data abstract value and the data table carried in the synchronization request information, determines a second data abstract value of the received data table through a Hash algorithm, judges whether the first data abstract value and the second data abstract value are consistent, and if so, performs import operation on the data table.
The technical scheme has the following advantages or beneficial effects:
in the application, when a data synchronization initiator initiates data synchronization request information, a data table to be synchronized is firstly acquired, then a first data abstract value of the data table is calculated, and the first data abstract value and the data table are carried in the data synchronization request information and sent to a data synchronization receiver. And the data synchronization receiver receives the data synchronization request information, and calculates a second data abstract value of the received data table after acquiring the first data abstract value and the data table carried in the data synchronization request information. When the first data abstract value is judged to be consistent with the second data abstract value, the data table sent by the data synchronization initiator is completely consistent with the data table received by the data synchronization receiver, and then the data table is imported, so that the synchronization consistency of the data tables is strictly ensured.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic diagram of a cloud multi-application data synchronization process provided in the present application;
fig. 2 is a diagram of a multi-application data synchronization architecture on the cloud according to the present application;
FIG. 3 is a schematic structural diagram of a synchronization device provided in the present application;
FIG. 4 is a schematic diagram of a two-dimensional planar infrastructure architecture provided by the present application;
FIG. 5 is a data synchronization principle architecture diagram provided herein;
FIG. 6 is a flow chart of data synchronization provided herein;
fig. 7 is a schematic structural diagram of a cloud multi-application data synchronization apparatus provided in the present application;
fig. 8 is a schematic diagram of a server structure provided in the present application.
Detailed Description
To make the purpose and embodiments of the present application clearer, the following will clearly and completely describe the exemplary embodiments of the present application with reference to the attached drawings in the exemplary embodiments of the present application, and it is obvious that the described exemplary embodiments are only a part of the embodiments of the present application, and not all of the embodiments.
It should be noted that the brief descriptions of the terms in the present application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of the present application. These terms should be understood in their ordinary and customary meaning unless otherwise indicated.
The terms "first," "second," "third," and the like in the description and claims of this application and in the above-described drawings are used for distinguishing between similar or analogous objects or entities and not necessarily for describing a particular sequential or chronological order, unless otherwise indicated. It is to be understood that the terms so used are interchangeable under appropriate circumstances.
The terms "comprises" and "comprising," as well as any variations thereof, are intended to cover a non-exclusive inclusion, such that a product or device that comprises a list of elements is not necessarily limited to all of the elements explicitly listed, but may include other elements not expressly listed or inherent to such product or device.
The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and/or software code that is capable of performing the functionality associated with that element.
Finally, it should be noted that: the above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.
The foregoing description, for purpose of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the embodiments to the precise forms disclosed above. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles and the practical application, to thereby enable others skilled in the art to best utilize the embodiments and various embodiments with various modifications as are suited to the particular use contemplated.
Fig. 1 is a schematic diagram of a cloud multi-application data synchronization process provided in the present application, where the process includes the following steps:
s101: the data synchronization method comprises the steps that a data synchronization initiator acquires a data table to be synchronized, a first data abstract value of the data table is determined through a Hash algorithm, the first data abstract value and the data table are stored in data synchronization request information, and the data synchronization request information is sent to a data synchronization receiver.
S102: and the data synchronization receiver receives the data synchronization request information, acquires the first data abstract value and the data table carried in the data synchronization request information, determines a second data abstract value of the received data table through a Hash algorithm, judges whether the first data abstract value is consistent with the second data abstract value, and if so, performs import operation on the data table.
The cloud multi-application data synchronization method is applied to the servers in the cloud multi-application architecture, the cloud multi-application architecture comprises the servers, and data synchronization requirements exist among the servers. And taking a server for initiating data synchronization as a data synchronization initiator and taking a target server for data synchronization as a data synchronization receiver.
After acquiring the data table to be synchronized, the data synchronization initiator performs hash calculation on the data table to be synchronized to obtain a first data abstract value, then carries the first data abstract value and the data table to be synchronized in data synchronization request information, and sends the data synchronization request information to the data synchronization receiver. After receiving the data synchronization request information, the data synchronization receiver analyzes the data synchronization request information to obtain a first data abstract value and a data table carried in the data synchronization request information, then performs hash calculation on the received data table to obtain a second data abstract value, and judges whether the first data abstract value and the second data abstract value are consistent, if so, the data synchronization is completed by performing import operation on the data table, otherwise, data import is not performed.
In the application, in order to ensure the security of data synchronization, after the data synchronization initiator carries the first data digest value and the data table to be synchronized in the data synchronization request information, a digital signature is added to the data synchronization request information through a digital signature algorithm. After receiving the data synchronization request information, the data synchronization receiver firstly checks the data synchronization request information. If the signature verification passes, analyzing the data synchronization request information to obtain a first data abstract value and a data table carried in the data synchronization request information, then carrying out Hash calculation on the received data table to obtain a second data abstract value, and judging whether the first data abstract value is consistent with the second data abstract value, if so, carrying out import operation on the data table to complete data synchronization, otherwise, not carrying out data import. And if the verification is not passed, the subsequent process of analyzing the data synchronization request information is not carried out.
In the application, when a data synchronization initiator initiates data synchronization request information, a data table to be synchronized is firstly acquired, then a first data abstract value of the data table is calculated, and the first data abstract value and the data table are carried in the data synchronization request information and sent to a data synchronization receiver. And the data synchronization receiver receives the data synchronization request information, and calculates a second data abstract value of the received data table after acquiring the first data abstract value and the data table carried in the data synchronization request information. When the first data abstract value is judged to be consistent with the second data abstract value, the data table sent by the data synchronization initiator is completely consistent with the data table received by the data synchronization receiver, and then the data table is imported, so that the consistency of the data table synchronization is strictly ensured.
In order to further ensure consistency of data synchronization, in the present application, the data synchronization initiator determines a data table to be synchronized, determines a first data digest value of the data table through a hash algorithm, and stores the first data digest value and the data table in data synchronization request information includes:
judging whether the current data synchronization service scene is a dependent service scene, if so, acquiring a dependent data table corresponding to the data table to be synchronized, determining a first data abstract value of the data table and a third data abstract value of the dependent data table through a hash algorithm, and storing the first data abstract value, the third data abstract value and the data table in data synchronization request information.
Before sending a data synchronization request message to a data synchronization receiver, a data synchronization initiator firstly judges whether a current data synchronization service scene is a dependent service scene, if not, performs data synchronization according to the scheme, and if the current data synchronization service scene is the dependent service scene, acquires a dependent data table corresponding to a data table to be synchronized according to a predetermined dependency relationship between the data tables. And performing hash calculation on the data table to be synchronized to obtain a first data abstract value, performing hash calculation on the dependent data table to obtain a third data abstract value, and storing the first data abstract value, the third data abstract value and the data table in the data synchronization request information.
For a dependent service scene, the data synchronization receiving party receives the synchronization request information, acquires the first data abstract value and the data table carried in the synchronization request information, determines a second data abstract value of the received data table through a hash algorithm, and judges whether the first data abstract value and the second data abstract value are consistent, if so, the importing operation on the data table comprises the following steps:
the data synchronization receiving party receives the synchronization request information and acquires the first data abstract value, the third data abstract value and the data table carried in the synchronization request information;
determining an imported dependent data table corresponding to the data table according to a preset import condition;
determining a second data digest value of the received data table and a fourth data digest value of the imported dependent data table through a hash algorithm;
and judging whether the first data abstract value is consistent with the second data abstract value, and whether the third data abstract value is consistent with the fourth data abstract value, and if so, importing the data table.
In the application, for a dependent service scene, a data synchronization receiving party receives data synchronization request information, and determines an imported dependent data table corresponding to the data table according to a preset import condition after acquiring the first data abstract value, the third data abstract value and the data table carried in the data synchronization request information through analysis; and then carrying out hash calculation on the imported dependent data table to obtain a fourth data abstract value, and carrying out hash calculation on the data table to obtain a second data abstract value.
If the first data abstract value and the second data abstract value are consistent, the current synchronous data table is consistent, and if the third data abstract value and the fourth data abstract value are consistent, the dependent data table corresponding to the current synchronous data table is consistent. And under the condition that the current synchronous data tables are consistent and the dependent data tables corresponding to the current synchronous data tables are consistent, carrying out import operation on the data tables to finish data synchronization, otherwise, not importing the data tables.
In order to further ensure consistency of data synchronization, before the importing operation is performed on the data table, the method further includes:
judging whether the imported data has the covered data corresponding to the data table, if so, backing up the covered data, storing the backed-up data to a specified address, deleting the covered data in the imported data, and importing the data table.
In one case, if the data synchronization receiver passes the verification of the data table, when a network fault or an import exception occurs in the process of importing the data table, the locally imported data is incomplete, when the data table needs to be imported again, the covered data exists in the database of the data synchronization receiver, and at this time, the covered data needs to be backed up and stored to the designated address. When the covered data is requested according to the service requirement, the covered data can be directly acquired to the designated address and corresponding service is carried out. And after the data to be covered is backed up and stored to the designated address, the data synchronization receiver deletes the data to be covered in the database, and then the data table is imported. Thereby ensuring the consistency of data synchronization.
The method further comprises the following steps:
and after the data table is imported, feeding back prompt information that the data are synchronized to the data synchronization initiator.
After the data synchronization receiver performs import operation on the data table, the prompt information that the data is synchronized can be fed back to the data synchronization initiator in a message form. So that the data synchronization initiator obtains the state of data synchronization.
The method further comprises the following steps:
and if the first data abstract value is judged to be inconsistent with the second data abstract value, returning the data table to the data synchronization initiator, and sending prompt information of abnormal synchronization data to a manager of the data synchronization initiator.
And if the first data abstract value is judged to be inconsistent with the second data abstract value, not performing data synchronization, namely not importing the data table. And when the first data abstract value and the second data abstract value are inconsistent, the synchronous data are abnormal, the data table is returned to the data synchronization initiator, and prompt information of the abnormal synchronous data is sent to a manager of the data synchronization initiator in a message form. So that the data synchronization initiator obtains the abnormal state of the data synchronization, and the abnormal problem is already existed.
The method further comprises the following steps:
and if the third data abstract value is judged to be inconsistent with the fourth data abstract value, returning the data table to the data synchronization initiator, and sending prompt information of the dependent abnormal synchronization task to a manager of the data synchronization initiator.
And if the third data abstract value is judged to be inconsistent with the fourth data abstract value, the data synchronization is not carried out, namely the data table is not imported. And if the third data abstract value and the fourth data abstract value are inconsistent, the dependent data synchronization abnormity is shown, the data sheet is returned to the data synchronization initiator, and prompt information of the dependent synchronization task abnormity is sent to a manager of the data synchronization initiator in a message form. So that the data synchronization initiator obtains the abnormal state of the data synchronization, and the abnormal problem is already existed.
The cloud multi-application data synchronization process provided by the present application is explained in detail below.
1. The cloud multi-application data synchronization is realized based on an API (application programming interface), each processing link of the data synchronization is processed in a pure asynchronous execution mode, and the corresponding time of the API is millisecond level.
2. The data synchronization method and the data synchronization system can realize multithreading of the data synchronization initiator and send data synchronization request information to a plurality of data synchronization receivers, the utilization rate of the CPU is high, multithreading concurrent processing is adopted in the data synchronization processing links by taking the database table as a unit, the service processing concurrency is high, and CPU computing resources can be fully utilized.
3. In order to control the overall processing time, the single data synchronization volume is controlled within a certain threshold range, for example, within 100M, and requests are sent for more than 100M. The record number of a single table supports the configuration of 'upper limit of number', and is sent in sub-messages. Therefore, the application in the operation period occupies a limited memory, and data transmission adopts stream transmission and is a processing process of the limited memory.
4. When the data synchronization request information is transmitted, the security of data transmission is improved through a digital signature algorithm, and the data transmission is transmitted by using an HTTPS network, so that the security of data transmission is further improved. And the overall data abstract is checked, so that the consistency of data synchronization is ensured.
The application uses the multi-center application data synchronization designed by the cloud native technology, adopts a two-dimensional plane micro-service architecture, implants an irreversible safe data verification technology taking a data signature algorithm as a core in a management plane, and realizes the requirements of high data synchronization consistency and data safety of a batch processing system under the multi-center hot standby architecture by a method of supporting two layers of verification factor calculation in the data plane, and meanwhile, the method has decoupling with an application system, elastic expansion and contraction capacity and wide applicability. The screening of high-dispersion transaction elements in financial transaction data and the summarization and verification technology of irreversible safety data of the transaction elements ensure high consistency and data safety of multi-center data synchronization.
Fig. 2 is a diagram of a multi-application data synchronization architecture on the cloud, which includes scheduling of a data synchronization field, service of the data synchronization field, and each service system. Each business system comprises a database and a synchronization device independent of the configuration of the database.
Fig. 3 is a schematic structural diagram of a synchronization apparatus provided in the present application, in which the synchronization apparatus is mainly divided into four working modules, namely, a configuration management module, a multi-center management module, a data synchronization module, and a task monitoring module.
The synchronization device provided by the application uses an HTTP general API communication protocol, and realizes a low-coupling data synchronization function applied to the cloud by means of a data security verification technology. Specific functions include two-dimensional plane central management and data synchronization functions supporting dynamic scaling, and specific working principles are described in detail below.
Fig. 4 is a schematic diagram of a two-dimensional plane infrastructure architecture provided by the present application, as shown in fig. 4, including a console mesh console and various service systems. Each service system comprises a synchronization device. The dotted lines in fig. 4 constitute the management plane and the solid lines constitute the data plane. An irreversible safe data verification technology taking a data signature algorithm as a core is implanted in a management plane, and two layers of verification factor calculation are supported in the data plane.
The multi-center management module is configured as follows:
(1) A central registry and a central status table.
The central registry provides configuration for the following information: center ID, machine ID, IP, PORT. And a central state table, namely, MACHINE _ ST, is added, active detection of the state of the central line is supported, and the central state table can record the connection state between the centers.
(2) And (4) centrally establishing a link and selecting the link.
Communication with other centers uses HTTP short links. When a synchronous message is sent to each center, a data synchronization request is sent to one machine in the center by using load balancing strategies such as polling or appointed machines.
(3) A unified data synchronization API.
a. A state detection interface:
GET, with null parameters, returns text body { code: "0000", msg: "}, an exception fills in a specific error code and cause.
b. Center identification query interface:
GET, with null parameters, returns text body { code: "0000", msg: ", centering: "master", exception is as above.
c. Data synchronous receiving interface:
and the POST requests the message body message to comprise { eventid: xxx, dataTaskId: xxx, params: LISTMAP, multimedia attachment format upload data file, answer is same as a, normal answer only indicates received.
d. The remote center response interface:
the POST, the request message body message includes { code: "0000", msg: ", eventid: xxx, datataskId: xxx, answer is the same as a.
The data sync field scheduling is as follows:
scheduling by using the job scheduling system in a field, wherein the specific field execution condition needs to be processed according to the following requirements:
a. dependent traffic patterns have performed successfully.
b. The corresponding service data has been executed.
c. The data synchronization tasks corresponding to the current payment date are successful (for limiting abnormal scenes such as redo and the like and avoiding the same-kind data synchronization conflict).
The function of each session is essentially the invocation of a data task.
a. And defining data synchronization content through data synchronization task configuration.
b. And filling parameters required by the corresponding task.
c. The device initiates data synchronization by calling the proposal.
The data synchronization module is configured as follows:
fig. 5 is a schematic diagram of the data synchronization principle provided in the present application.
(1) The tasks are defined as follows:
a. task ID: allocations were developed for presentation only.
b. And (3) task description: the definition was developed for presentation only.
c. The task type is as follows: data transmission and file transmission.
d. Data configuration: [ { database, entity name, table name, condition: { [ { field, type, operator, value } - ], SQL _ WHERE } }, the. And (4) occupying variables, filling actual values when initiating, and assembling the device into condition filling parameters according to configured primary keys in a unified manner.
e. And (3) configuring the financial field columns, wherein for the target table needing to be derived in use, the financial field columns are derived according to the d configuration conditions, and the financial field columns are specially derived according to the number of data in a set threshold range, wherein the set threshold range is 5000-6000 data, 5000-7000 data and the like. After the summary, the whole system calculates the data abstract value by using SHA256 hash algorithm, and stores the data abstract value and the corresponding data table in the data synchronization request information through hash mapping, and sends the data abstract value and the corresponding data table to the backup center, and the financial field column belongs to the attribute which must be configured. For the synchronization of non-financial data, time information, state information and other fields which are frequently changed can be appointed to achieve the same effect.
(2) Data export and initiation synchronization are as follows:
a. the file floor is generated firstly by the exported data.
d. Data is derived in two ways: a configuration condition field, or a conditional SQL.
c. A secure data digest is generated. For the data synchronization task with dependence, waiting for the completion of the dependence task, before the data synchronization is started, firstly exporting financial field columns according to the number of data in a set threshold range, after the data synchronization is summarized, calculating a data abstract value by using an SHA256 hash algorithm as an abstract factor A, then calculating a data abstract of the newly added data as an abstract factor B, and placing the two calculated abstract values as an abstract pair in a data synchronization request message; for a data synchronization task which is simple and independent, financial field columns are specially exported according to the number of data in a set threshold range, after summary, a SHA256 hash algorithm is used for calculating a data abstract value, the abstract value B and an EMPTY abstract factor A _ EMPTY are matched into a pair, and the data abstract value B and a corresponding data file are stored in data synchronization request information through hash mapping and are sent to a standby center.
(3) Data import is as follows:
a. after receiving the import file, the import file is landed. For a data synchronization task with a dependent task, before importing, the data information abstract of the stock is calculated according to the importing condition. If the data is EMPTY, the digest factor setting a _ EMPTY does not need to be repeatedly calculated, and if data exists, the digest value a' is calculated.
b. The data synchronization containment degree of the standby center: the dependent conditions can be configured to satisfy the conditions for data coverage.
c. And backing up the covered data before importing. And deleting the data after the backup is finished, and then inserting the synchronous data.
d. Import and data summarization factor calculation: in order to avoid the situation that data is imported in a large-transaction mode of batch submission because the synchronous data volume is overlarge, the import of each piece of data is automatically submitted to be put in storage, if the import link of backup is abnormal, the backup data is used for backing up the data of the current import operation, and the data consistency of an application layer is realized. And after the import is finished, calculating a data information abstract factor B' of the imported incremental data according to the same method.
e. And (4) safety verification: comparing the summary factors A 'and B' calculated by the receiving side with A and B in the message, and considering the data to be consistent only when the two are equal. If A is consistent with A 'but B is not equal to B', the synchronous data has problems, all currently imported data are backed off, and error information is fed back to operation and maintenance personnel. If A is not consistent with A 'but B is equal to B', the data synchronization is consistent, but the dependent synchronization task has a fault, all the currently imported data are backed off, and error information is fed back to operation and maintenance personnel.
Fig. 6 is a flow chart of data synchronization provided in the present application, and as shown in fig. 6, one data synchronization includes 5 specific data synchronization steps. The method comprises the steps that technical parameters of two sides of data synchronization are loaded respectively from front to back, a data synchronization initiator (a main center) waits for operation on a service to complete database operation, a data synchronization device generates a data file from incremental data of the current operation after the completion and data security verification summary information, the data file is sent to a data synchronization receiver (a standby center) of the data synchronization through an HTTP protocol after the completion, the data synchronization receiver falls to the ground after receiving a request, specific import operation is started after the data security verification is completed, and finally a processing result is fed back to the data synchronization initiator.
Fig. 7 is a schematic structural diagram of a cloud multi-application data synchronization apparatus provided in the present application, where the apparatus includes:
the sending module 71 is configured to acquire a data table to be synchronized, determine a first data digest value of the data table through a hash algorithm, store the first data digest value and the data table in data synchronization request information, and send the data synchronization request information to a data synchronization receiving party;
a receiving module 72, configured to receive the data synchronization request information, obtain the first data digest value and the data table carried in the data synchronization request information, determine, through a hash algorithm, a second data digest value of the received data table, determine whether the first data digest value and the second data digest value are consistent, and if so, perform an import operation on the data table.
The sending module 71 is specifically configured to determine whether a current data synchronization service scenario is a dependent service scenario, if so, obtain a dependent data table corresponding to the data table to be synchronized, determine a first data digest value of the data table and a third data digest value of the dependent data table through a hash algorithm, and store the first data digest value, the third data digest value, and the data table in data synchronization request information.
The receiving module 72 is specifically configured to receive the synchronization request information by the data synchronization receiving party, and acquire the first data digest value, the third data digest value, and the data table carried in the synchronization request information; determining an imported dependent data table corresponding to the data table according to a preset import condition; determining a second data digest value of the received data table and a fourth data digest value of the imported dependent data table through a hash algorithm; and judging whether the first data abstract value is consistent with the second data abstract value, and whether the third data abstract value is consistent with the fourth data abstract value, and if so, performing import operation on the data table.
The receiving module 72 is further configured to determine whether there is covered data corresponding to the data table in the imported data, if so, backup the covered data, store the backed up data to a specified address, delete the covered data in the imported data, and perform an import operation on the data table.
The device further comprises:
the first feedback module 73 is configured to feed back, after the import operation is performed on the data table, prompt information that data is synchronized to the data synchronization initiator.
The device further comprises:
and a second feedback module 74, configured to, if it is determined that the first data digest value is inconsistent with the second data digest value, rollback the data table to the data synchronization initiator, and send a prompt message that the synchronized data is abnormal to a manager of the data synchronization initiator.
The device further comprises:
a third feedback module 75, configured to, if it is determined that the third data digest value is inconsistent with the fourth data digest value, rollback the data table to the data synchronization initiator, and send a prompt message indicating that a dependent synchronization task is abnormal to a manager of the data synchronization initiator.
The present application also provides a server, as shown in fig. 8, including: the system comprises a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302 and the memory 303 complete mutual communication through the communication bus 304;
the memory 303 has stored therein a computer program which, when executed by the processor 301, causes the processor 301 to perform any of the above method steps.
The communication bus mentioned in the above server may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface 302 is used for communication between the above-described server and other devices.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Alternatively, the memory may be at least one memory device located remotely from the aforementioned processor.
The Processor may be a general-purpose Processor, including a central processing unit, a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an application specific integrated circuit, a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like.
The present application further provides a computer-storage-readable storage medium having stored therein a computer program executable by a server, the program, when executed on the server, causing the server to perform any of the above method steps when executed.
While the preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all alterations and modifications as fall within the scope of the application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (16)
1. A method for synchronizing multi-application data on a cloud, the method comprising:
a data synchronization initiator acquires a data table to be synchronized, determines a first data abstract value of the data table through a Hash algorithm, stores the first data abstract value and the data table in data synchronization request information, and sends the data synchronization request information to a data synchronization receiver;
and the data synchronization receiver receives the data synchronization request information, acquires the first data abstract value and the data table carried in the data synchronization request information, determines a second data abstract value of the received data table through a Hash algorithm, judges whether the first data abstract value is consistent with the second data abstract value, and if so, performs import operation on the data table.
2. The method of claim 1, wherein the data synchronization initiator determining a data table to be synchronized, determining a first data digest value of the data table through a hashing algorithm, and depositing the first data digest value and the data table in a data synchronization request message comprises:
judging whether the current data synchronization service scene is a dependent service scene, if so, acquiring a dependent data table corresponding to the data table to be synchronized, determining a first data abstract value of the data table and a third data abstract value of the dependent data table through a hash algorithm, and storing the first data abstract value, the third data abstract value and the data table in data synchronization request information.
3. The method according to claim 2, wherein the data synchronization receiving party receives the synchronization request message, obtains the first data digest value and the data table carried therein, determines a second data digest value of the received data table through a hash algorithm, and determines whether the first data digest value and the second data digest value are consistent, and if so, importing the data table includes:
the data synchronization receiving party receives the data synchronization request information and acquires the first data abstract value, the third data abstract value and the data table carried in the data synchronization request information;
determining an imported dependent data table corresponding to the data table according to a preset import condition;
determining a second data digest value of the received data table and a fourth data digest value of the imported dependent data table through a hash algorithm;
and judging whether the first data abstract value is consistent with the second data abstract value, and whether the third data abstract value is consistent with the fourth data abstract value, and if so, importing the data table.
4. The method of claim 1, wherein prior to the importing operation on the data table, the method further comprises:
judging whether the imported data has the covered data corresponding to the data table or not, if so, backing up the covered data, storing the backed-up data to a specified address, deleting the covered data in the imported data, and importing the data table.
5. The method of claim 1, wherein the method further comprises:
and after the data table is imported, feeding back prompt information that the data are synchronized to the data synchronization initiator.
6. The method of claim 3, wherein the method further comprises:
and if the first data abstract value is judged to be inconsistent with the second data abstract value, returning the data table to the data synchronization initiator, and sending prompt information of abnormal synchronization data to a manager of the data synchronization initiator.
7. The method of claim 3, wherein the method further comprises:
and if the third data abstract value is judged to be inconsistent with the fourth data abstract value, returning the data table to the data synchronization initiator, and sending a prompt message of the dependent abnormal synchronization task to a manager of the data synchronization initiator.
8. An apparatus for synchronizing multi-application data on a cloud, the apparatus comprising:
the sending module is used for acquiring a data table to be synchronized, determining a first data abstract value of the data table through a Hash algorithm, storing the first data abstract value and the data table in data synchronization request information, and sending the data synchronization request information to a data synchronization receiver;
and the receiving module is used for receiving the data synchronization request information, acquiring the first data abstract value and the data table carried in the data synchronization request information, determining a second data abstract value of the received data table through a Hash algorithm, judging whether the first data abstract value is consistent with the second data abstract value, and if so, importing the data table.
9. The apparatus according to claim 8, wherein the sending module is specifically configured to determine whether a current data synchronization service scenario is a dependent service scenario, and if so, obtain a dependent data table corresponding to the data table to be synchronized, determine a first data digest value of the data table and a third data digest value of the dependent data table through a hash algorithm, and store the first data digest value, the third data digest value, and the data table in the data synchronization request message.
10. The apparatus according to claim 9, wherein the receiving module is specifically configured to receive the synchronization request message by the data synchronization receiving side, and obtain the first data digest value, the third data digest value, and the data table carried therein; determining an imported dependent data table corresponding to the data table according to a preset import condition; determining a second data digest value of the received data table and a fourth data digest value of the imported dependent data table through a hash algorithm; and judging whether the first data abstract value is consistent with the second data abstract value, and whether the third data abstract value is consistent with the fourth data abstract value, and if so, importing the data table.
11. The apparatus according to claim 8, wherein the receiving module is further configured to determine whether there is covered data corresponding to the data table in the imported data, and if so, backup the covered data and save the backed-up data to a specified address, delete the covered data in the imported data, and perform an import operation on the data table.
12. The apparatus of claim 8, wherein the apparatus further comprises:
and the first feedback module is used for feeding back prompt information that the data are synchronized to the data synchronization initiator after the data table is imported.
13. The apparatus of claim 10, wherein the apparatus further comprises:
and the second feedback module is used for returning the data table to the data synchronization initiator and sending prompt information of abnormal synchronous data to a manager of the data synchronization initiator if the first data abstract value is judged to be inconsistent with the second data abstract value.
14. The apparatus of claim 10, wherein the apparatus further comprises:
and the third feedback module is used for returning the data table to the data synchronization initiator and sending prompt information of the dependent abnormal synchronization task to a manager of the data synchronization initiator if the third data abstract value is judged to be inconsistent with the fourth data abstract value.
15. The server is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing the communication between the processor and the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 7 when executing a program stored in the memory.
16. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211345921.2A CN115643271A (en) | 2022-10-31 | 2022-10-31 | Method, device, server and medium for synchronizing multi-application data on cloud |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211345921.2A CN115643271A (en) | 2022-10-31 | 2022-10-31 | Method, device, server and medium for synchronizing multi-application data on cloud |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115643271A true CN115643271A (en) | 2023-01-24 |
Family
ID=84946774
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211345921.2A Pending CN115643271A (en) | 2022-10-31 | 2022-10-31 | Method, device, server and medium for synchronizing multi-application data on cloud |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115643271A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116738507A (en) * | 2023-08-14 | 2023-09-12 | 中科亿海微电子科技（苏州）有限公司 | Chip authentication method |
-
2022
- 2022-10-31 CN CN202211345921.2A patent/CN115643271A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116738507A (en) * | 2023-08-14 | 2023-09-12 | 中科亿海微电子科技（苏州）有限公司 | Chip authentication method |
| CN116738507B (en) * | 2023-08-14 | 2023-11-10 | 中科亿海微电子科技（苏州）有限公司 | Chip authentication method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105245373B (en) | A kind of container cloud platform system is built and operation method | |
| CN110096340B (en) | Timing task processing method and device | |
| CN109614209B (en) | Task processing method, application server and system | |
| CN110825420A (en) | Configuration parameter updating method, device, equipment and storage medium for distributed cluster | |
| CN104011701A (en) | Content delivery network | |
| CN111314479A (en) | Data processing method and equipment | |
| CN106713391B (en) | Session information sharing method and sharing system | |
| CN113268471B (en) | Method, proxy connection pool, system, device and medium for processing distributed transaction | |
| CN108446172B (en) | Data calling method and device, computer equipment and storage medium | |
| CN111600957A (en) | File transmission method, device and system and electronic equipment | |
| US10721335B2 (en) | Remote procedure call using quorum state store | |
| CN108140035B (en) | Database replication method and device for distributed system | |
| CN111400041A (en) | Server configuration file management method and device and computer readable storage medium | |
| CN112698838A (en) | Multi-cloud container deployment system and container deployment method thereof | |
| CN113703997A (en) | Bidirectional asynchronous communication middleware system integrating multiple message agents and implementation method | |
| CN115643271A (en) | Method, device, server and medium for synchronizing multi-application data on cloud | |
| CN105373563B (en) | Database switching method and device | |
| WO2020006896A1 (en) | Balance monitoring method, apparatus, computer device, and storage medium | |
| CN116384993B (en) | Method and system for realizing high consistency of order payment state based on cloud payment center | |
| CN115098528B (en) | Service processing method, device, electronic equipment and computer readable storage medium | |
| CN111190754A (en) | Block chain event notification method and block chain system | |
| CN104657240B (en) | The Failure Control method and device of more kernel operating systems | |
| CN112711466B (en) | Hanging affair inspection method and device, electronic equipment and storage medium | |
| CN109034768B (en) | Financial reconciliation method, apparatus, computer device and storage medium | |
| US20210240698A1 (en) | Asynchronous remote calls with undo data structures |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |