CN112948491A - Data synchronization method and device, terminal equipment and computer readable storage medium - Google Patents
Data synchronization method and device, terminal equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN112948491A CN112948491A CN202110220513.3A CN202110220513A CN112948491A CN 112948491 A CN112948491 A CN 112948491A CN 202110220513 A CN202110220513 A CN 202110220513A CN 112948491 A CN112948491 A CN 112948491A
- Authority
- CN
- China
- Prior art keywords
- execution data
- data
- statement
- message queue
- downstream component
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/544—Buffers; Shared memory; Pipes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present application relates to the field of data synchronization processing, and in particular, to a method, an apparatus, a terminal device and a computer-readable storage medium for data synchronization, where the method includes the following steps: making tangent points on the bottom layer of the database, and configuring the base class of the bottom layer of the database into a tangent plane class according to the tangent points; when business operation is received, if the calling method of the business operation triggers the section class, the SQL statement contained in the business operation is intercepted; converting the SQL statement into a standard statement in a preset format through a statement interface, and acquiring execution data representing the SQL statement in the standard statement; and pushing the execution data to a message queue so that the downstream component acquires the execution data from the message queue and synchronizes the execution data to the local of the downstream component. According to the method and the system, the transaction codes are injected into the method class needing to operate the database base class based on the facet class in a non-invasive mode, the upstream and downstream component data are synchronized based on the transaction codes, and the data synchronization efficiency is improved.
Description
Technical Field
The present application relates to the field of data synchronization processing, and in particular, to a method, an apparatus, a terminal device and a computer-readable storage medium for data synchronization.
Background
In the big data era, the importance of data synchronization is increasing day by day, and data sharing between systems is also becoming more and more important. The existing data synchronization mode includes data synchronization by using a database trigger, and the data synchronization mode needs to establish a large number of triggers for data synchronization, and the large number of triggers can affect the execution of a service system, and needs to spend more resources to maintain the triggers for data synchronization, so that the triggering deployment of the data synchronization is too complex, and the efficiency of the data synchronization is too low; in addition, the data synchronization method includes data synchronization by using full-table copy, which is to periodically empty the destination data source and then copy the data of the source data to the destination data source in a full disk manner, and this data synchronization method is very inefficient and cannot quickly and comprehensively perform triggering of data synchronization. In summary, the data synchronization has a problem of low synchronization efficiency.
Disclosure of Invention
The present application mainly aims to provide a data synchronization method, an apparatus, a terminal device and a storage medium, and aims to solve the problem of low efficiency of the current data synchronization.
In order to achieve the above object, the present application provides a method for data synchronization, including:
making tangent points on the bottom layer of the database, and configuring the base class of the bottom layer of the database into a tangent plane class according to the tangent points;
when business operation is received, if the calling method of the business operation triggers the section class, the SQL statement contained in the business operation is intercepted;
converting the SQL statement into a standard statement in a preset format through a statement interface, and acquiring execution data representing the SQL statement in the standard statement;
and pushing the execution data to a message queue so that the downstream component acquires the execution data from the message queue and synchronizes the execution data to the local of the downstream component.
Further, before pushing the execution data to the message queue, the method further includes:
acquiring a primary key identifier in the execution data;
obtaining a cache associated with the primary key identifier;
and storing the execution data into a cache associated with the primary key identification.
Further, the pushing the execution data to a message queue includes:
and according to a preset time interval, sequentially acquiring the execution data from the cache according to the time sequence of the execution data stored in the cache, and then pushing the execution data to a message queue.
Further, after the converting the SQL statement into a standard statement in a preset format via a statement interface and acquiring the execution data representing the SQL statement in the standard statement, the method further includes:
acquiring the security level of the service operation;
configuring the storage duration of the execution data according to the security level;
and storing the execution data in a local storage system in a persistent mode, and configuring the storage time length corresponding to the execution data.
Further, after pushing the execution data to a message queue to enable a downstream component to obtain the execution data from the message queue and synchronize to a local downstream component, the method further includes:
when receiving the message that the downstream component completes data synchronization, counting down the storage duration;
and deleting the data which are synchronized in the local storage system after the countdown reaches the storage time length.
Further, after the obtaining of the execution data characterizing the SQL statement in the standard statement, the method further includes:
acquiring an operation type contained in the execution data;
matching a corresponding message queue according to the operation type;
the pushing the execution data to a message queue includes:
and pushing the execution data to a message queue corresponding to the operation type.
Further, after pushing the execution data to a message queue to enable a downstream component to obtain the execution data from the message queue and synchronize to a local downstream component, the method further includes:
controlling a downstream component to convert the execution data into an execution operation instruction;
and executing the local data of the downstream component correspondingly based on the execution operation instruction.
The present application further provides a device based on data synchronization, including:
section configuration module: the method comprises the steps of making tangent points on a database bottom layer, and configuring a base class of the database bottom layer into a tangent plane class according to the tangent points;
section triggering module: the method comprises the steps that when business operation is received, if a calling method of the business operation triggers the section class, SQL sentences contained in the business operation are intercepted;
a data conversion module: the SQL statement is converted into a standard statement in a preset format through a statement interface, and the execution data representing the SQL statement in the standard statement is obtained;
a data synchronization module: the system is used for pushing the execution data to a message queue so that a downstream component acquires the execution data from the message queue and synchronizes the execution data to the local of the downstream component.
The application also provides a terminal device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the data synchronization method according to any one of the above items when executing the computer program.
The present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method for data synchronization of any of the above.
The application provides a method for triggering an upstream component and a downstream component to execute data synchronization based on a tangent plane type, a tangent point is made on a database bottom layer based on a tangent plane-oriented programming, a base class of the database bottom layer is configured into a tangent plane type, then a transaction code of the tangent plane type is defined, the base class of the database bottom layer is configured into the tangent plane type, the transaction code can be injected into any method class needing to operate the database base class without invasion, if a calling method of business operation of a current component business layer triggers the tangent plane type, the transaction code of the tangent plane type is executed, the transaction code comprises an SQL statement intercepting the business operation, the SQL statement is converted into a standard statement in a preset format through a statement interface, execution data representing the SQL statement in the standard statement is obtained, and the SQL statement is determined to execute a certain operation on a certain row of data of a certain table in a database according to the execution data, and pushing the execution data to a message queue, controlling a downstream component to acquire the execution data from the message queue, and synchronizing the execution data to the local downstream component, so that the data synchronization of the upstream component and the downstream component is realized, the creation of a trigger and the writing of a subsequent execution logic of the trigger are not required to be performed on each table of the database, the deployment operation of triggering the data synchronization can be simplified, the cost of maintaining the trigger in the database table is reduced, and the efficiency of data synchronization is improved.
Drawings
FIG. 1 is a schematic flow chart diagram illustrating an embodiment of a method for data synchronization according to the present application;
FIG. 2 is a schematic diagram of another embodiment of the data synchronization method of the present application;
FIG. 3 is a schematic diagram illustrating an embodiment of a data synchronization apparatus according to the present application;
fig. 4 is a block diagram illustrating a structure of an embodiment of a terminal device according to the present application.
The implementation, functional features and advantages of the objectives of the present application will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
Referring to fig. 1, an embodiment of the present application provides a method for data synchronization, which includes steps S10-S40, and the detailed description of the steps of the method for data synchronization is as follows.
And S10, making tangent points on the bottom layer of the database, and configuring the base class of the bottom layer of the database into tangent planes according to the tangent points.
The embodiment is applied to a scenario in which an upstream and downstream component of a project synchronizes a large amount of data in a database, in the whole project, a tangent point is made on the bottom layer of the database based on an AOP, wherein AOP (aspect organized programming) refers to facet-Oriented programming, and through the facet-Oriented programming, a required logic code can be added to a service layer without affecting the logic of an existing service layer, that is, a logic code for performance detection or transaction control is added to a class method for triggering the tangent point by the service layer, in the embodiment, a Spring AOP is used to make a tangent point on the bottom layer of the database, and a base class BaseDAO of the bottom layer of the database is configured as a facet class according to the tangent point, the facet class includes a definition part of the trigger logic for the tangent point and also includes a definition part of a transaction code after triggering the facet class, and the transaction code can run through the whole project or through a specific function module, the base class of the bottom layer of the database is configured into a tangent class through AOP, so that the transaction code defined by the tangent class can be injected into any method class needing to operate the base class of the database without invasion, and the synchronization of the upstream and downstream component data is realized according to the transaction code.
S20, when a business operation is received, if the calling method of the business operation triggers the section class, the SQL statement contained in the business operation is intercepted.
In this embodiment, when a business layer executes a business operation, different business operations trigger different message events, and after receiving a message event generated by the business operation of the business layer, it is determined whether the call method of the business operation triggers the facet class, specifically, it is determined whether a method class corresponding to the call method of the business operation inherits a base class of the database bottom layer, that is, whether a method class corresponding to the call method of the business operation directly or indirectly uses the base class of the database bottom layer, if yes, the call method of the business operation triggers the facet class, and after the call method of the business operation triggers the facet class, a transaction code included in the facet class is executed, which is equivalent to injecting a transaction code of the facet class into the call method of the business operation, and the transaction code first intercepts an SQL statement included in the business operation, the SQL statement comprises operations of adding, deleting, changing and checking data in the database. The business operation is a business operation generated by the current component.
S30, converting the SQL statement into a standard statement in a preset format through a statement interface, and acquiring the execution data representing the SQL statement in the standard statement.
In this embodiment, after intercepting the SQL statements contained in the business operation, in order to synchronize the execution information of the SQL statements to the downstream components, and reduces the transmission of useless information, converts the SQL statement into a standard statement with a preset format through a statement interface, the statement interface can eliminate the interference information in the SQL statement, then extracting the information required by the specification, combining the extracted information into a preset format as a standard statement, therefore, each SQL statement included in the business operation is converted into a standard statement, and the converted standard statement has a preset format, so that the execution data representing the SQL statement in the standard statement can be quickly acquired, the execution data comprises four data of table name, primary key ID, operation type, update field and value, according to the execution data, the SQL statement can be determined to execute a certain operation on a certain row of data of a certain table in the database.
And S40, pushing the execution data to a message queue so that the downstream component acquires the execution data from the message queue and synchronizes the execution data to the local downstream component.
In this embodiment, after the current component executes a business operation, and after the execution data of an SQL statement representing the business operation is obtained through the above steps, the execution data needs to be notified to a downstream component, and the business operation of the upstream component is not affected, at this time, the current component is equivalent to the upstream component, in this embodiment, the execution data is pushed to a message queue, the message queue preferentially uses a Kafka message queue, after the execution data is pushed to the message queue, the downstream component can consume the execution data from the message queue, that is, the downstream component is controlled to obtain the execution data from the message queue, and synchronize the execution data to the local of the downstream component, so as to achieve synchronization of data of the upstream component and the downstream component, that is, the upstream component completes the business operation, and when the upstream component randomly triggers the service operation of the section class of the bottom base class of the database, the execution data corresponding to the service operation is synchronized to the downstream component, so that the data synchronization of the upstream component and the downstream component is rapidly completed.
The embodiment provides a method for triggering an upstream component and a downstream component to execute data synchronization based on a section-oriented program, which includes making a tangent point on a database bottom layer based on the section-oriented program, configuring a base class of the database bottom layer as a section class, then defining a transaction code of the section class, configuring the base class of the database bottom layer as the section class, injecting the transaction code to any method class needing to operate the database base class noninvasively, if a calling method of a business operation of a current component business layer triggers the section class, executing the transaction code of the section class, wherein the transaction code comprises an SQL statement intercepting the business operation, converting the SQL statement into a standard statement in a preset format via a statement interface, acquiring the execution data representing the SQL statement in the standard statement, and determining that the SQL statement executes a certain operation on a certain row of data of a certain table in the database according to the execution data, and pushing the execution data to a message queue, controlling a downstream component to acquire the execution data from the message queue, and synchronizing the execution data to the local downstream component, so that the data synchronization of the upstream component and the downstream component is realized, the creation of a trigger and the writing of a subsequent execution logic of the trigger are not required to be performed on each table of the database, the cost of maintaining the trigger in the database table is reduced, and the efficiency of data synchronization is improved.
In one embodiment, as shown in fig. 2, before pushing the execution data to the message queue, the method further includes:
s31: acquiring a primary key identifier in the execution data;
s32: obtaining a cache associated with the primary key identifier;
s33: and storing the execution data into a cache associated with the primary key identification.
In this embodiment, after obtaining the execution data representing the SQL statement, in order to smoothly notify a downstream component of each time of the execution data according to a chronological order, the execution data needs to be effectively classified and sorted, and specifically, a primary key identifier in the execution data is obtained, in one implementation, a primary key identifier in the execution data is obtained by using an annotation @ Options (key property "id" or key column ") or a < selectKey > tag, that is, the primary key identifier is returned after the execution of the SQL statement is completed, the primary key identifier represents a unique identifier of a database table operated by the execution data, then a plurality of caches storing the execution data are created, different caches are respectively associated with different primary key identifiers, then the execution data of the same primary key identifier is stored in the same cache, that is, the execution data is respectively stored in the caches associated with the primary key identifiers according to the primary key identifiers, the synchronization of the execution data corresponding to the same primary key identification can be unified, and the ordering of data synchronization is improved.
In an application scenario, a set function of Redis is introduced to realize that the execution data is respectively stored in the caches associated with the primary key identifiers according to the primary key identifiers, specifically, X cache sets are configured to store the execution data to be synchronized, the sum of ASCII of the primary key identifiers (represented by the primary key ID) is divided by X, and the remainder is put into the corresponding cache, so that the execution data of the same primary key identifier can be stored in the same cache.
In one embodiment, said pushing said execution data to a message queue comprises:
and according to a preset time interval, sequentially acquiring the execution data from the cache according to the time sequence of the execution data stored in the cache, and then pushing the execution data to a message queue.
In this embodiment, since the execution data is stored in the caches associated with the primary key identifiers respectively, and each execution data is stored in the caches according to the time sequence, in order to ensure the sequence of the execution data, different execution data corresponding to the same primary key identifier is sequentially obtained from one cache according to the time sequence in which the execution data is stored in the caches according to the preset time interval, and then the execution data is pushed to the message queue, so that only one piece of return data of the same primary key identifier is pushed to the message queue each time, so as to notify the downstream component of the execution data, since the message queue is stored according to the primary key identifiers, the execution data of the same primary key identifier is stored in the same cache, and only one piece of execution data is pushed once by pushing the execution data of the same primary key identifier, so that the sequence of the execution data notification to the downstream component is ensured, the correctness of data synchronization of the upstream component and the downstream component is ensured.
In one embodiment, after the converting the SQL statement into a standard statement in a preset format via a statement interface and acquiring execution data representing the SQL statement in the standard statement, the method further includes:
acquiring the security level of the service operation;
configuring the storage duration of the execution data according to the security level;
and storing the execution data in a local storage system in a persistent mode, and setting the storage time length corresponding to the execution data.
In this embodiment, after converting an SQL statement into a standard statement in a preset format through a statement interface, and acquiring execution data representing the SQL statement in the standard statement, because different execution data needs to be synchronized with different timeliness, part of the execution data needs to be synchronized within a first preset time, and part of the execution data needs to be synchronized within a second preset time, in order to avoid loss of the execution data, after the execution data is acquired, the execution data is persistently stored in a local storage system, so as to improve protection of the execution data, and avoid data loss due to system downtime, further, while the execution data is persistently stored in the local storage system, a security level of a service operation, that is, a security level of a service operation corresponding to the execution data is acquired, and then a storage time of the execution data is configured according to the security level, after the execution data is stored in the local storage system in a persistent mode, and the storage time length corresponding to the execution data is configured, the situation that the resources of the local storage system are occupied by excessive execution data is avoided, and therefore the utilization rate of the resources of the local storage system is improved.
In one embodiment, after pushing the execution data to a message queue so that a downstream component acquires the execution data from the message queue and synchronizes to a local of the downstream component, the method further includes:
when receiving the message that the downstream component completes data synchronization, counting down the storage duration;
and after the countdown reaches the storage time length, deleting the data which are synchronized in the local storage system.
In this embodiment, after the downstream component is controlled to obtain the execution data from the message queue and synchronize the execution data to the local of the downstream component, the upstream component needs to pay attention to the completion degree of the data synchronization of the downstream component, when a message that the data synchronization of the downstream component is completed is received, the execution data stored in the local storage system is counted down, the counted down time is the storage time, after the counted down time reaches the storage time, the corresponding execution data in the local storage system is deleted, so as to avoid the situation that the storage time of the execution data is set too short, and the downstream component does not complete the data synchronization, so as to delete the corresponding execution data in the local storage system, thereby improving the disaster tolerance measure for the execution data, and delete the execution data in the local storage system at regular time, so as to vacate the storage resource in the local storage system, the utilization rate of resources is improved.
In one embodiment, after the obtaining of the execution data characterizing the SQL statement in the standard statement, the method further includes:
acquiring an operation type contained in the execution data;
matching a corresponding message queue according to the operation type;
the pushing the execution data to a message queue includes:
and pushing the execution data to a message queue corresponding to the operation type.
In this embodiment, after the execution data representing the SQL statement in the standard statement is obtained, operation type information included in the execution data is also obtained, where the operation type includes operation types classified by adding, deleting, modifying, and searching data, and then a corresponding message queue is matched according to the operation type. In this embodiment, different message queues are matched for different operation types of execution data, and then the execution data is pushed to a message queue corresponding to the operation type, so as to control a downstream component to acquire the execution data of a corresponding type from the message queue. In one embodiment, when the downstream component is a read data component, the downstream component (read data component) is controlled to acquire execution data from a read message queue and then synchronize to the downstream component; and when the downstream component is the data writing component, controlling the downstream component to acquire the execution data from the write message queue and then synchronizing to the downstream component. By dividing the execution data of different operation types, the functional division can be carried out on the message queue, the exclusive use of the message queue is ensured, and the performance of the message queue with the special processing function is improved. In addition, different downstream components can acquire the execution data matched with the downstream component from different message queues, and the efficiency of data synchronization of the downstream components can be effectively improved.
In one embodiment, after pushing the execution data to a message queue so that a downstream component acquires the execution data from the message queue and synchronizes to a local of the downstream component, the method further includes:
controlling a downstream component to convert the execution data into an execution operation instruction;
and executing the local data of the downstream component correspondingly based on the execution operation instruction.
In this embodiment, after controlling the downstream component to synchronize the execution data to the local downstream component, the execution data includes four data, i.e. a table name, a primary key ID, an operation type, an update field, and a value, and then the downstream component is controlled to convert the execution data into an execution operation instruction according to the four data, namely, the SQL statement which can be executed by the downstream component is converted, and then the local data of the downstream component is executed correspondingly based on the execution operation instruction, when the downstream component executes the operation instruction, if the downstream component deploys the data synchronization method scheme of this embodiment, and then notifies the downstream component of the SQL statement based on the data synchronization method scheme of this embodiment, the downstream component at this time is converted into the current component, i.e., the upstream component, then notifies the downstream component of the execution data according to the steps of the above-described embodiments, completing the data synchronization over the entire data link.
Referring to fig. 3, the present application provides an apparatus for data synchronization, including:
section configuration module 10: the method comprises the steps of making tangent points on a database bottom layer, and configuring a base class of the database bottom layer into a tangent plane class according to the tangent points;
section triggering module 20: the method comprises the steps that when business operation is received, if a calling method of the business operation triggers the section class, SQL sentences contained in the business operation are intercepted;
the data conversion module 30: the SQL statement is converted into a standard statement in a preset format through a statement interface, and the execution data representing the SQL statement in the standard statement is obtained;
the data synchronization module 40: the system is used for pushing the execution data to a message queue so that a downstream component acquires the execution data from the message queue and synchronizes the execution data to the local of the downstream component.
As described above, it can be understood that the components of the data synchronization apparatus proposed in the present application can implement the functions of any one of the above-described data synchronization methods.
In one embodiment, the data synchronization module 40 further performs:
acquiring a primary key identifier in the execution data;
obtaining a cache associated with the primary key identifier;
and storing the execution data into a cache associated with the primary key identification.
In one embodiment, the data synchronization module 40 further performs:
and according to a preset time interval, sequentially acquiring the execution data from the cache according to the time sequence of the execution data stored in the cache, and then pushing the execution data to a message queue.
In one embodiment, the apparatus further comprises a data storage module to:
acquiring the security level of the service operation;
configuring the storage duration of the execution data according to the security level;
and storing the execution data in a local storage system in a persistent mode, and configuring the storage time length corresponding to the execution data.
In one embodiment, the data storage module further performs:
when receiving the message that the downstream component completes data synchronization, counting down the storage duration;
and after the countdown reaches the storage time length, deleting the data which are synchronized in the local storage system.
In one embodiment, the data conversion module 30 further performs:
acquiring an operation type contained in the execution data;
matching a corresponding message queue according to the operation type;
the data synchronization module 40 further performs:
and pushing the execution data to a message queue corresponding to the operation type.
In one embodiment, the data synchronization module 40 further performs:
controlling a downstream component to convert the execution data into an execution operation instruction;
and executing the local data of the downstream component correspondingly based on the execution operation instruction.
Referring to fig. 4, a terminal device is further provided in the embodiment of the present application, and an internal structure of the terminal device may be as shown in fig. 4. The terminal equipment comprises a processor, a memory, a network interface, a display device and an input device which are connected through a system bus. The network interface of the terminal equipment is used for connecting and communicating with an external terminal through a network. The display device of the terminal equipment is used for displaying the interactive page. The input device of the terminal equipment is used for receiving the input of a user. The computer designed processor is used to provide computational and control capabilities. The memory of the terminal device includes a nonvolatile storage medium. The non-volatile storage medium stores an operating system, a computer program, and a database. The database of the terminal device is used for storing original data. The computer program is executed by a processor to implement a method of data synchronization.
The processor executes the method for data synchronization, which includes: based on AOP, making tangent points on the bottom layer of the database, and configuring the base class of the bottom layer of the database into a tangent plane class according to the tangent points; when business operation is received, if the calling method of the business operation triggers the section class, the SQL statement contained in the business operation is intercepted; converting the SQL statement into a standard statement in a preset format through a statement interface, and acquiring execution data representing the SQL statement in the standard statement; and pushing the execution data to a message queue, and controlling a downstream component to acquire the execution data from the message queue and synchronize the execution data to the local of the downstream component. The terminal equipment provides a method for triggering an upstream component and a downstream component to execute data synchronization based on a tangent plane type, a tangent point is made on a database bottom layer based on a tangent plane-oriented programming, a base class of the database bottom layer is configured into a tangent plane type, then a transaction code of the tangent plane type is defined, the transaction code can be injected into any method class needing to operate the database base class without invasion by configuring the base class of the database bottom layer into the tangent plane type, if a calling method of business operation of a current component business layer triggers the tangent plane type, the transaction code of the tangent plane type is executed, the transaction code comprises an SQL statement intercepting the business operation, the SQL statement is converted into a standard statement in a preset format through a presentation interface, execution data representing the SQL statement in the standard statement is obtained, and the SQL statement is determined to execute a certain data operation on a certain row of a certain table in the database according to the execution data, and pushing the execution data to a message queue, controlling a downstream component to acquire the execution data from the message queue, and synchronizing the execution data to the local downstream component, so that the data synchronization of the upstream component and the downstream component is realized, the creation of a trigger and the writing of a subsequent execution logic of the trigger are not required to be performed on each table of the database, the cost of maintaining the trigger in the database table is reduced, and the efficiency of data synchronization is improved.
An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, the computer program, when executed by the processor, implementing a method for data synchronization, including the steps of: based on AOP, making tangent points on the bottom layer of the database, and configuring the base class of the bottom layer of the database into a tangent plane class according to the tangent points; when business operation is received, if the calling method of the business operation triggers the section class, the SQL statement contained in the business operation is intercepted; converting the SQL statement into a standard statement in a preset format through a statement interface, and acquiring execution data representing the SQL statement in the standard statement; and pushing the execution data to a message queue, and controlling a downstream component to acquire the execution data from the message queue and synchronize the execution data to the local of the downstream component. The computer readable storage medium provides a method for triggering an upstream component and a downstream component to execute data synchronization based on a tangent plane class, a tangent point is made on a database bottom layer based on a tangent plane-oriented programming, a base class of the database bottom layer is configured into a tangent plane class, then a transaction code of the tangent plane class is defined, the transaction code can be injected into any method class needing to operate the database base class without invasion by configuring the base class of the database bottom layer into the tangent plane class, if a calling method of a business operation of a current component business layer triggers the tangent plane class, the transaction code of the tangent plane class is executed, the transaction code comprises an SQL statement intercepting the business operation, the SQL statement is converted into a standard statement in a preset format through a statement interface, execution data representing the SQL statement in the standard statement is obtained, and the SQL statement is determined to execute a certain data of a certain table in the database according to the execution data, and pushing the execution data to a message queue, controlling a downstream component to acquire the execution data from the message queue, and synchronizing the execution data to the local downstream component, so that the data synchronization of the upstream component and the downstream component is realized, the creation of a trigger and the writing of a subsequent execution logic of the trigger are not required to be performed on each table of the database, the cost of maintaining the trigger in the database table is reduced, and the efficiency of data synchronization is improved.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium provided herein and used in the examples may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double-rate SDRAM (SSRSDRAM), Enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application, or which are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.
Claims (10)
1. A method of data synchronization, comprising:
making tangent points on the bottom layer of the database, and configuring the base class of the bottom layer of the database into a tangent plane class according to the tangent points;
when business operation is received, if the calling method of the business operation triggers the section class, the SQL statement contained in the business operation is intercepted;
converting the SQL statement into a standard statement in a preset format through a statement interface, and acquiring execution data representing the SQL statement in the standard statement;
and pushing the execution data to a message queue so that the downstream component acquires the execution data from the message queue and synchronizes the execution data to the local of the downstream component.
2. The method of data synchronization of claim 1, wherein before pushing the execution data to a message queue, further comprising:
acquiring a primary key identifier in the execution data;
obtaining a cache associated with the primary key identifier;
and storing the execution data into a cache associated with the primary key identification.
3. The method of data synchronization of claim 2, wherein pushing the execution data to a message queue comprises:
and according to a preset time interval, sequentially acquiring the execution data from the cache according to the time sequence of the execution data stored in the cache, and then pushing the execution data to a message queue.
4. The method according to claim 1, wherein the converting the SQL statement into a standard statement in a preset format via a statement interface, and after acquiring the execution data representing the SQL statement in the standard statement, further comprises:
acquiring the security level of the service operation;
configuring the storage duration of the execution data according to the security level;
and storing the execution data in a local storage system in a persistent mode, and configuring the storage time length corresponding to the execution data.
5. The method of claim 4, wherein after pushing the execution data to a message queue to enable a downstream component to obtain the execution data from the message queue and synchronize to a local of the downstream component, the method further comprises:
when receiving the message that the downstream component completes data synchronization, counting down the storage duration;
and after the countdown reaches the storage time length, deleting the data which are synchronized in the local storage system.
6. The method according to claim 1, wherein after the obtaining of the execution data characterizing the SQL statement in the standard statement, the method further comprises:
acquiring an operation type contained in the execution data;
matching a corresponding message queue according to the operation type;
the pushing the execution data to a message queue includes:
and pushing the execution data to a message queue corresponding to the operation type.
7. The method of claim 1, wherein after pushing the execution data to a message queue to enable a downstream component to obtain the execution data from the message queue and synchronize to a local of the downstream component, the method further comprises:
controlling a downstream component to convert the execution data into an execution operation instruction;
and executing the local data of the downstream component correspondingly based on the execution operation instruction.
8. An apparatus for data synchronization, comprising:
section configuration module: the method comprises the steps of making tangent points on a database bottom layer, and configuring a base class of the database bottom layer into a tangent plane class according to the tangent points;
section triggering module: the method comprises the steps that when business operation is received, if a calling method of the business operation triggers the section class, SQL sentences contained in the business operation are intercepted;
a data conversion module: the SQL statement is converted into a standard statement in a preset format through a statement interface, and the execution data representing the SQL statement in the standard statement is obtained;
a data synchronization module: the system is used for pushing the execution data to a message queue so that a downstream component acquires the execution data from the message queue and synchronizes the execution data to the local of the downstream component.
9. A terminal device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method for data synchronization according to any of claims 1 to 7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method for data synchronization according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110220513.3A CN112948491B (en) | 2021-02-26 | 2021-02-26 | Method, device, terminal equipment and computer readable storage medium for data synchronization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110220513.3A CN112948491B (en) | 2021-02-26 | 2021-02-26 | Method, device, terminal equipment and computer readable storage medium for data synchronization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112948491A true CN112948491A (en) | 2021-06-11 |
| CN112948491B CN112948491B (en) | 2023-09-22 |
Family
ID=76246662
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110220513.3A Active CN112948491B (en) | 2021-02-26 | 2021-02-26 | Method, device, terminal equipment and computer readable storage medium for data synchronization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112948491B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114979039A (en) * | 2022-06-21 | 2022-08-30 | 国网电商科技有限公司 | Data processing method and device |
| CN116455917A (en) * | 2023-06-15 | 2023-07-18 | 上海观安信息技术股份有限公司 | Data synchronization method and device, electronic equipment and computer readable storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4398272A (en) * | 1972-05-04 | 1983-08-09 | Schlumberger Technology Corporation | Computerized truck instrumentation system |
| CN106682119A (en) * | 2016-12-08 | 2017-05-17 | 杭州销冠网络科技有限公司 | System and method for asynchronous data synchronization on basis of http service aspect and log system |
| CN108200219A (en) * | 2018-03-13 | 2018-06-22 | 广东欧珀移动通信有限公司 | Method of data synchronization, device, server and storage medium |
| WO2020147392A1 (en) * | 2019-01-16 | 2020-07-23 | 平安科技（深圳）有限公司 | Method and system for data synchronization between databases |
-
2021
- 2021-02-26 CN CN202110220513.3A patent/CN112948491B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4398272A (en) * | 1972-05-04 | 1983-08-09 | Schlumberger Technology Corporation | Computerized truck instrumentation system |
| CN106682119A (en) * | 2016-12-08 | 2017-05-17 | 杭州销冠网络科技有限公司 | System and method for asynchronous data synchronization on basis of http service aspect and log system |
| CN108200219A (en) * | 2018-03-13 | 2018-06-22 | 广东欧珀移动通信有限公司 | Method of data synchronization, device, server and storage medium |
| WO2020147392A1 (en) * | 2019-01-16 | 2020-07-23 | 平安科技（深圳）有限公司 | Method and system for data synchronization between databases |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114979039A (en) * | 2022-06-21 | 2022-08-30 | 国网电商科技有限公司 | Data processing method and device |
| CN116455917A (en) * | 2023-06-15 | 2023-07-18 | 上海观安信息技术股份有限公司 | Data synchronization method and device, electronic equipment and computer readable storage medium |
| CN116455917B (en) * | 2023-06-15 | 2023-09-22 | 上海观安信息技术股份有限公司 | Data synchronization method and device, electronic equipment and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112948491B (en) | 2023-09-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107688500B (en) | Distributed task processing method, device, system and equipment | |
| CN108228814B (en) | Data synchronization method and device | |
| CN111859470B (en) | Business data chaining method and device | |
| CN111310427A (en) | Service data configuration processing method and device, computer equipment and storage medium | |
| CN109389299B (en) | Workflow process deployment method, workflow process deployment device, computer equipment and storage medium | |
| CN112948491B (en) | Method, device, terminal equipment and computer readable storage medium for data synchronization | |
| CN113590386B (en) | Disaster recovery method, system, terminal device and computer storage medium for data | |
| CN111045806A (en) | Method and system for realizing delay message queue | |
| CN113760513A (en) | Distributed task scheduling method, device, equipment and medium | |
| CN111784318A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN108196831B (en) | Construction method and device of business system | |
| CN109614271B (en) | Method, device, equipment and storage medium for controlling consistency of multiple cluster data | |
| CN111562952B (en) | Dynamic loading method and dynamic loading device for double-core intelligent ammeter management unit | |
| CN111984659A (en) | Data updating method and device, computer equipment and storage medium | |
| CN116841758A (en) | Workflow task processing method, device, computer equipment and storage medium | |
| CN113407551A (en) | Data consistency determining method, device, equipment and storage medium | |
| CN114296789B (en) | Business processing method, device, equipment and storage medium based on full-process configuration | |
| CN116048609A (en) | Configuration file updating method, device, computer equipment and storage medium | |
| CN111722881B (en) | Resource expansion method, system and device of container cloud platform | |
| KR101888131B1 (en) | Method for Performing Real-Time Changed Data Publish Service of DDS-DBMS Integration Tool | |
| CN107844340B (en) | Method and system for data acquisition and file downloading | |
| CN113868138A (en) | Method, system, equipment and storage medium for acquiring test data | |
| CN113590643A (en) | Data synchronization method, device, equipment and storage medium based on dual-track database | |
| CN112164433A (en) | Hospital background data processing method and system | |
| CN109542609A (en) | A kind of method, apparatus withholding refund, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230828 Address after: No. 546, Luoyu Road, Hongshan District, Wuhan, Hubei Province, 430000 Applicant after: HUBEI CENTRAL CHINA TECHNOLOGY DEVELOPMENT OF ELECTRIC POWER Co.,Ltd. Address before: 518000 Room 202, block B, aerospace micromotor building, No.7, Langshan No.2 Road, Xili street, Nanshan District, Shenzhen City, Guangdong Province Applicant before: Shenzhen LIAN intellectual property service center Effective date of registration: 20230828 Address after: 518000 Room 202, block B, aerospace micromotor building, No.7, Langshan No.2 Road, Xili street, Nanshan District, Shenzhen City, Guangdong Province Applicant after: Shenzhen LIAN intellectual property service center Address before: 518000 Room 201, building A, No. 1, Qian Wan Road, Qianhai Shenzhen Hong Kong cooperation zone, Shenzhen, Guangdong (Shenzhen Qianhai business secretary Co., Ltd.) Applicant before: PING AN PUHUI ENTERPRISE MANAGEMENT Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |