CN114328738A - Data processing method and device, electronic equipment and readable storage medium - Google Patents
Data processing method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN114328738A CN114328738A CN202111650609.XA CN202111650609A CN114328738A CN 114328738 A CN114328738 A CN 114328738A CN 202111650609 A CN202111650609 A CN 202111650609A CN 114328738 A CN114328738 A CN 114328738A
- Authority
- CN
- China
- Prior art keywords
- data
- target data
- target
- data table
- cache queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The disclosure provides a data processing method and device, electronic equipment and a readable storage medium, and relates to the technical field of computers. The method comprises the following steps: acquiring a data dictionary of a target data table from a source database; judging whether the number of the data dictionaries stored in the preset cache queue at the current moment is less than a first threshold value or not; if not, deleting the data dictionary stored at the tail of the preset cache queue at the current moment from the preset cache queue, and storing the data dictionary of the target data table to the head of the preset cache queue; and synchronizing at least one piece of target data of the target data table to a target database by analyzing a data dictionary of the target data table at the head of the preset cache queue. In the process, when the number of the data dictionaries in the preset cache queue is not less than the first threshold value, the data dictionary at the tail of the preset cache queue is deleted from the preset cache queue, so that the memory occupied by the data dictionaries is not increased without limit, and the memory overflow is prevented.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method and apparatus, an electronic device, and a readable storage medium.
Background
The information technology is rapidly developed, and the database plays a very important role in the storage and processing of information data. When data synchronization is performed between databases, a conventional method is to acquire a log file of a source database for data synchronization software, and load all data dictionaries in the source database into an internal memory according to the log file of the source database, thereby implementing data synchronization.
However, when data synchronization is performed by the above conventional method, the memory occupied by the data dictionary will increase continuously with the progress of the data synchronization process, which may cause memory overflow.
Disclosure of Invention
In order to solve the technical problem, the present disclosure provides a data processing method, an apparatus, an electronic device, and a readable storage medium.
In a first aspect, an embodiment of the present disclosure provides a data processing method, including:
acquiring a data dictionary of a target data table from a source database, wherein the target data table comprises at least one piece of target data;
judging whether the number of the data dictionaries stored in the preset cache queue at the current moment is less than a first threshold value or not;
if not, deleting the data dictionary stored at the tail of the preset cache queue at the current moment from the preset cache queue, and storing the data dictionary of the target data table to the head of the preset cache queue;
and synchronizing at least one piece of target data in the target data table to a target database by analyzing the data dictionary of the target data table stored at the head of the preset cache queue.
As an optional implementation manner of this embodiment of the present disclosure, before acquiring the data dictionary of the target data table in the source database, the method further includes:
judging whether a data dictionary of the target data table is stored in the preset cache queue or not;
if so, moving the data dictionary of the target data table to the head of the preset cache queue;
and if not, acquiring a data dictionary of the target data table from the source database.
As an optional implementation manner of the embodiment of the present disclosure, the obtaining a data dictionary of a target data table in a source database includes:
acquiring a log file of the source database in the source database, and acquiring the ID of the target data table through the log file of the source database;
and acquiring a data dictionary of the target data table in the source database based on the ID identification of the target data table.
As an optional implementation manner of the embodiment of the present disclosure, the method further includes:
and if the number of the data dictionaries stored in the preset cache queue at the current moment is less than a first threshold value, storing the data dictionary of the target data table to the head of the preset cache queue, and not deleting the data dictionary stored at the tail of the preset cache queue at the current moment.
As an optional implementation manner of the embodiment of the present disclosure, the synchronizing at least one piece of target data in the target data table to the target database by analyzing the data dictionary stored at the head of the preset cache queue includes:
analyzing the data dictionary stored at the head of the preset cache queue to obtain data operation corresponding to the at least one piece of target data;
and synchronizing the at least one piece of target data to a target database based on the data operation corresponding to the at least one piece of target data.
As an optional implementation manner of the embodiment of the present disclosure, the synchronizing the at least one piece of target data to the target database based on the data operation corresponding to the at least one piece of target data includes:
constructing a synchronous statement based on the data operation corresponding to the at least one piece of target data;
synchronizing the at least one piece of target data to a target database by executing the synchronization statement.
As an optional implementation manner of the embodiment of the present disclosure, the storing the data dictionary of the target data table to the head of the preset buffer queue includes:
constructing a cache node corresponding to the target data table based on information included in the data dictionary of the target data table;
and storing the cache node corresponding to the target data table to the head of the preset cache queue.
In a second aspect, an embodiment of the present disclosure provides a data processing apparatus, including:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring a data dictionary of a target data table in a database, and the target data table comprises at least one piece of target data;
the judging module is used for judging whether the number of the data dictionaries stored in the preset cache queue at the current moment is less than a first threshold value or not;
the processing module is used for deleting the data dictionary stored at the tail of the preset cache queue at the current moment from the preset cache queue and storing the data dictionary of the target data table to the head of the preset cache queue if the number of the data dictionaries stored in the preset cache queue at the current moment is not less than a first threshold value;
and the synchronization module is used for synchronizing at least one piece of target data in the target data table to a target database by analyzing the data dictionary of the target data table stored at the head of the preset cache queue.
As an optional implementation manner of the embodiment of the present disclosure, the determining module is further configured to determine whether a data dictionary of the target data table is stored in the preset cache queue; if so, moving the data dictionary of the target data table to the head of the preset cache queue; and if not, acquiring a data dictionary of the target data table from the source database.
As an optional implementation manner of the embodiment of the present disclosure, the obtaining module is specifically configured to obtain a log file of a source database, and obtain an ID identifier of a data table corresponding to target data through the log file of the source database to obtain the ID identifier of the target data table; and acquiring a data dictionary of the target data table from the source database according to the ID of the target data table.
As an optional implementation manner of the embodiment of the present disclosure, the apparatus further includes: and the analysis module is used for storing the data dictionary of the target data table to the head of the preset cache queue if the number of the data dictionaries stored in the preset cache queue at the current moment is smaller than a first threshold value, and not deleting the data dictionary stored at the tail of the preset cache queue at the current moment.
As an optional implementation manner of the embodiment of the present disclosure, the synchronization module includes an analysis unit and an execution unit;
the analysis unit is used for analyzing the data dictionary stored at the head of the preset cache queue to obtain data operation corresponding to the at least one piece of target data;
the execution unit is used for synchronizing the at least one piece of target data to a target database based on the data operation corresponding to the at least one piece of target data.
As an optional implementation manner of the embodiment of the present disclosure, the execution unit is specifically configured to construct a synchronization statement based on a data operation corresponding to the at least one piece of target data; synchronizing the at least one piece of target data to a target database by executing the synchronization statement.
As an optional implementation manner of the embodiment of the present disclosure, the processing module is specifically configured to construct a cache node corresponding to the target data table based on information included in the data dictionary of the target data table; and storing the cache node corresponding to the target data table to the head of the preset cache queue.
In a third aspect, the present disclosure provides an electronic device comprising: a memory and a processor;
the memory is configured to store computer program instructions;
the processor is configured to execute the computer program instructions such that the electronic device implements the data processing method of any of the first aspects.
In a fourth aspect, the present disclosure provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the data processing method according to any one of the optional embodiments of the first aspect.
Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:
in the technical scheme provided by the embodiment of the disclosure, the data dictionary of the target data table is obtained from the source database, whether the number of the data dictionaries stored in the preset cache queue at the current moment is smaller than a first threshold value or not is judged, if not, the data dictionary stored at the tail of the preset cache queue at the current moment is deleted from the preset cache queue, the data dictionary of the target data table is stored to the head of the preset cache queue, and at least one piece of target data of the target data table is synchronized to the target database by analyzing the data dictionary of the target data table at the head of the preset cache queue. In the method provided by the embodiment of the disclosure, when the number of the data dictionaries in the preset buffer queue is not less than the first threshold, the data dictionary at the tail of the preset buffer queue is deleted from the preset buffer queue, so that the memory occupied by the data dictionary is not increased without limit, and the memory overflow is prevented. Therefore, abnormal phenomena such as jamming or automatic quitting in the data synchronization process are avoided.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present disclosure, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
FIG. 1 is a flow chart illustrating steps of a data processing method according to an embodiment of the present disclosure;
fig. 2 is a schematic diagram of a preset buffer queue in a data processing method according to an embodiment of the present disclosure;
FIG. 3 is a flow chart illustrating steps of a data processing method according to another embodiment of the present disclosure;
fig. 4 is a schematic diagram of a preset buffer queue in a data processing method according to another embodiment of the present disclosure;
FIG. 5 is a flow chart illustrating steps of a data processing method according to yet another embodiment of the present disclosure;
fig. 6 is a block diagram of a data processing apparatus according to an embodiment of the present disclosure;
fig. 7 is a block diagram of a data processing apparatus according to another embodiment of the present disclosure;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present disclosure.
Detailed Description
In order that the above objects, features and advantages of the present disclosure may be more clearly understood, aspects of the present disclosure will be further described below. It should be noted that the embodiments and features of the embodiments of the present disclosure may be combined with each other without conflict.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure, but the present disclosure may be practiced in other ways than those described herein; it is to be understood that the embodiments disclosed in the specification are only a few embodiments of the present disclosure, and not all embodiments.
Firstly, an application scenario of the scheme of the present disclosure is introduced: data synchronization between databases is a way to achieve data consistency and uniformity between databases, and when data synchronization is performed between databases, it is a common way to synchronize data in a source database to a target database by using a data synchronization tool. The method comprises the following steps that when a data synchronization tool is used for carrying out real-time data synchronization, three stages are divided, wherein the first stage comprises the step of carrying out initialization loading on stock data to obtain a data synchronization base point; the second stage comprises incremental data synchronization by taking a synchronization base point established by initializing data loading as a reference; and the third stage comprises comparing and checking the source data and the target data of the data synchronization periodically to confirm that no data is lost in the data synchronization process. The second stage and the third stage may be in a long-time parallel state, and the scheme provided by the embodiment of the disclosure may be applied to a synchronization process of the second stage on the incremental data.
The execution main body of the data processing method provided by the embodiment of the disclosure may be data synchronization software, and the data synchronization software may be installed on the electronic device provided by the disclosure, and the electronic device may be implemented in any software and/or hardware manner through the execution of the electronic device in the embodiment of the disclosure. By way of example, the electronic device may be, but is not limited to, a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), a smart watch, a smart bracelet, and other terminal devices, and the disclosure is not limited to a specific type of terminal device.
Based on the foregoing description, the data processing method provided by the present disclosure is explained in detail with reference to the drawings and application scenarios of the embodiments of the present disclosure.
Fig. 1 is a data processing method according to an embodiment of the present disclosure, which may include the following steps:
s110, acquiring a data dictionary of the target data table from the source database.
The target data table comprises at least one piece of target data to be synchronized, and in the incremental data synchronization stage, the target data can comprise all incremental data, when the incremental data is stored in one data table, the target data table is one, and when the incremental data is stored in a plurality of data tables, the target data table comprises a plurality of data tables. The data dictionary of the target data table comprises information of data items, data structures, data structure descriptions, data flow descriptions and compositions, data storage descriptions and compositions, processing procedures and the like in the target data table. A piece of data in the disclosed embodiments may refer to a data record, represented as a row of data in a data table.
The data dictionary for acquiring the target data table in the source database comprises a data dictionary for acquiring one target data table, and can also comprise a data dictionary for acquiring a plurality of target data tables.
Optionally, obtaining the data dictionary of the target data table in the source database may be implemented as follows: acquiring a log file of a source database in the source database, and acquiring an ID (identity) of a data table corresponding to target data through the log file of the source database to obtain an ID of the target data table; and acquiring a data dictionary of the target data table from the source database according to the ID of the target data table. The log file in the source database includes an online log or an archive log, and changes such as addition, deletion, and modification of data in the source database and information of a corresponding data table are recorded in the log file in the source database, so that the target data and the ID of the target data table corresponding to the target data can be determined in the log file in the source database.
Further, the table name or column name of the target data table may be obtained from the data dictionary of the target data table. Illustratively, a log file of a source database is acquired, an ID identifier of a target data table where target data of a current operation is acquired from the log file of the source database is 1, and a data dictionary with the ID identifier of 1 is determined according to the ID identifier of 1 of the target data table, that is, the data dictionary of the target data table with the ID identifier of 1.
S120, judging whether the number of the data dictionaries stored in the preset buffer queue at the current moment is smaller than a first threshold value.
The preset cache queue is a memory space of a data dictionary which is constructed in advance and used for storing a plurality of data tables, and in the preset cache queue, the storage mode of the data dictionaries can include: the data dictionaries are stored based on an LRU algorithm (least recently used), namely, the corresponding data dictionary is stored according to the time interval between the time when the last use of each data dictionary is finished and the current time: and sequencing the data dictionaries according to the sequence that the time interval between the time when the last use is finished and the current time is from small to large, wherein the first data dictionary row corresponding to the minimum time interval is positioned at the head of the preset cache queue, the second data dictionary row corresponding to the time interval is stored, and the like, and the last data dictionary row corresponding to the maximum time interval is positioned at the tail of the preset cache queue. Namely, the time interval between the time when the last use of the data dictionary positioned at the head of the preset buffer queue is finished and the current time is minimum, and the time interval between the time when the last use of the data dictionary positioned at the tail of the preset buffer queue is finished and the current time is maximum.
In addition, the storage mode of the plurality of data dictionaries may further include: the data dictionaries are stored based on an LFU algorithm (used less frequently recently), namely, the corresponding data dictionary is stored according to the use frequency of each data dictionary within preset time: the data dictionaries are sequenced from high to low according to the use frequency of each data dictionary within the preset time, the data dictionary with the highest use frequency within the preset time is arranged at the head of the preset cache queue, the data dictionary with the second use frequency is stored behind the data dictionary at the head of the preset cache queue, and so on, and the data dictionary with the lowest use frequency within the preset time is arranged at the last data dictionary at the tail of the preset cache queue.
In this embodiment, the first threshold, the length of the preset buffer queue, and the preset time are not specifically limited. The "use" of the data dictionary in the present embodiment includes: for example, to synchronize data in a data table corresponding to a data dictionary, if one or more pieces of data included in a data table a in a source database are synchronized to a target database within a preset time, the data dictionary of the data table a is used within the preset time.
And if the number of the data dictionaries stored in the preset cache queue at the current moment is less than a first threshold value, storing the data dictionary of the target data table to the head of the preset cache queue, and not deleting the data dictionary stored at the tail of the preset cache queue at the current moment. For example, in a specific implementation, a data dictionary of lruFirst pointing to the head of the preset cache queue, a data dictionary of lruLast pointing to the tail of the preset cache queue may be defined, and if it is assumed that lruFirst points to the data dictionary of table2 and lruLast points to the data dictionary of table3 before the data dictionary of target data table1 is saved to the head of the preset cache queue, lruFirst points to the data dictionary of table1 and lruLast still points to the data dictionary of table3 when the data dictionary of target data table1 is saved to the preset cache queue, so that the data dictionary of target data table1 is saved at the head of the preset cache queue.
If the number of the data dictionaries stored in the preset buffer queue at the current time is not less than the first threshold, the following step S130 is executed.
It should be noted that the first threshold is less than or equal to the maximum number of dictionaries that can be stored in the preset buffer queue.
In the first case, when the first threshold is less than the maximum number of dictionaries that can be stored in the preset cache queue, and the preset cache queue is not full, but the number of the stored data dictionaries reaches the first threshold, the data dictionary at the tail of the preset cache queue is deleted, the data dictionary at the second last in the preset cache queue is moved backward to the tail of the preset cache queue, the data dictionary at the third last in the preset cache queue is moved backward to the second last … … in the preset cache queue, that is, after the data dictionary at the tail of the preset cache queue at the current moment is deleted, the sequences of the remaining data dictionaries are sequentially moved backward, and the data dictionary of the target data table is stored to the head of the preset cache queue, so that the space of the preset cache queue is prevented from being completely occupied.
In the second case, when the first threshold is equal to the maximum number of dictionaries that can be stored in the preset buffer queue and the number of data dictionaries stored in the preset buffer queue at the present time reaches the maximum number of dictionaries that can be stored in the preset buffer queue, the data dictionary at the tail of the preset buffer queue is deleted, so that memory overflow in the data synchronization process is prevented.
S130, deleting the data dictionary stored at the tail of the preset buffer queue at the current moment from the preset buffer queue, and storing the data dictionary of the target data table to the head of the preset buffer queue.
Optionally, the storage form of the data dictionary in the preset cache queue may be a cache node, and each cache node stores a data dictionary of a data table. One implementation manner of saving the data dictionary of the target data table to the head of the preset buffer queue includes: constructing a cache node corresponding to the target data table based on information included in the data dictionary of the target data table; and storing the cache node corresponding to the target data table to the head of the preset cache queue. Illustratively, the queue order of the cache node storing the data dictionary of the target data table in the preset cache queue is the first bit, and the cache node located before the first bit cache node points to the cache node of the data dictionary of the target data table, so as to implement storing the data dictionary of the target data table at the head of the preset cache queue.
The data dictionary of the target data table comprises information of data items, data structures, data structure descriptions, data flow descriptions and compositions, data storage descriptions and compositions, processing procedures and the like in the target data table. Each cache node in the cache queue stores a data dictionary corresponding to the data table and information (before and after) for realizing the bidirectional linked list, and the information of the bidirectional linked list is used for appointing a previous cache node and a next cache node of each cache node.
With reference to fig. 2, a description will be given of an implementation in which, when the number of cache nodes (data dictionaries) in the preset cache queue at the present time is not less than the first threshold, the data dictionary stored at the tail of the preset cache queue is deleted from the preset cache queue, and the cache node corresponding to the target data table is stored at the head of the preset cache queue. Illustratively, the preset cache queue shown in fig. 2 includes a cache node 1, a cache node 2, a cache node 3, and a cache node 4, where the cache node 1 is located at a first bit of the preset cache queue, a second bit of the preset cache queue is the cache node 2, a third bit of the preset cache queue is the cache node 3, and a last bit of the preset cache queue is the cache node 4. When a target cache node (target data dictionary) corresponding to the target data table is stored in the preset cache queue, the cache node 4 at the tail of the preset cache queue is deleted from the preset cache queue, the cache node 1, the cache node 2 and the cache node 3 sequentially move backwards one bit, the cache node 3 is positioned at the tail of the preset cache queue, and the target cache node is stored to the head of the preset cache queue.
In a specific implementation, lruFirst may be defined to point to a cache node at the head of the preset cache queue, and lruLast may be defined to point to a cache node at the tail of the preset cache queue. After deleting the cache node at the tail of the preset cache queue and storing the target cache node in the preset cache queue, lruFirst points to the target cache node, and lruLast points to the cache node 3.
And caching and/or eliminating the data dictionaries in the preset cache queue according to the relation between the number of the data dictionaries stored in the preset cache queue and the first threshold value, so that the length of the preset cache queue can be dynamically maintained, the preset cache queue is prevented from overflowing a memory, and the normal operation of data synchronization is ensured. In addition, the frequently accessed data dictionary is moved to the front position (lruFirst) of the preset buffer queue, and the frequently accessed data dictionary information is gradually moved to the rear position (lruLast) of the preset buffer queue, so that each time a new data dictionary of a target data table needs to be buffered, if the space of the preset buffer queue (indexedlucache) is insufficient or reaches the first threshold value, the data dictionary (lruLast) at the tail of the queue is preferentially deleted, that is, the frequently accessed data dictionary is not used. Therefore, the data dictionary of unused and idle data table data is eliminated every time. The hit efficiency of the data dictionary of the target data table stored in the preset cache queue is improved to the maximum extent on the premise of ensuring limited resources (memory).
S140, synchronizing at least one piece of target data in the target data table to the target database by analyzing the data dictionary of the target data table stored at the head of the preset buffer queue.
One way to implement this is to parse the data dictionary of the target data table, combine the log file of the source database, obtain at least one piece of transaction information of the target data in the source database, and synchronize at least one piece of target data to the target database based on the at least one piece of transaction information of the target data in the source database.
In the technical scheme provided by the embodiment of the disclosure, the data dictionary of the target data table is obtained from the source database, whether the number of the data dictionaries stored in the preset cache queue at the current moment is smaller than a first threshold value or not is judged, if not, the data dictionary stored at the tail of the preset cache queue at the current moment is deleted from the preset cache queue, the data dictionary of the target data table is stored to the head of the preset cache queue, and at least one piece of target data of the target data table is synchronized to the target database by analyzing the data dictionary of the target data table at the head of the preset cache queue. In the data processing method provided by the embodiment of the disclosure, when the number of the data dictionaries in the preset buffer queue is not less than the first threshold, the data dictionary at the tail of the preset buffer queue is deleted from the preset buffer queue, so that the memory occupied by the data dictionary is not increased without limit, and the memory overflow is prevented. Therefore, abnormal phenomena such as jamming or automatic quitting in the data synchronization process are avoided.
Fig. 3 is a flowchart of steps of a data processing method according to another embodiment of the present disclosure, and on the basis of the embodiment shown in fig. 1, further before step S110, the method further includes:
s310, judging whether a data dictionary of the target data table is stored in a preset buffer queue.
If yes, go to step S320, otherwise go to step S110. The explanation and explanation of step S110 refer to the embodiment shown in fig. 1, and are not repeated here.
And S320, moving the data dictionary of the target data table to the head of a preset buffer queue.
Specifically, if the data dictionary of the target data table is stored in the preset cache queue, the data dictionary of the target data table is not obtained from the source database. And if the data dictionary of the target data table is positioned at the head of the preset cache queue, directly analyzing the data dictionary of the target data table stored at the head of the preset cache queue, and synchronizing at least one piece of target data in the target data table to the target database. And if the position of the data dictionary of the target data table in the preset cache queue is not the head of the preset cache queue, moving the data dictionary of the target data table to the head of the preset cache queue, then analyzing, and synchronizing at least one piece of target data in the target data table to the target database based on the analysis result.
With reference to fig. 4, taking an example that each data dictionary is stored in a preset cache queue in the form of a cache node, a process of moving a data dictionary of a target data table to a head of the preset cache queue is described, where 4 cache nodes are stored in the preset cache queue shown in fig. 4, and a data dictionary storing one data table is stored in each cache node, which is sequentially a cache node a, a cache node B, a cache node C, and a cache node D from the head to the tail of the queue according to a sequence, and if the data dictionary of the target data table is stored in the cache node C, the cache node C is moved to the head of the preset cache queue, and other cache nodes are sequentially moved backward.
For example, in a specific implementation, lruFirst may be defined to point to a cache node at the head of a preset cache queue, and lruLast may be defined to point to a cache node at the tail of the preset cache queue.
Based on the above description, what is deleted from the preset buffer queue is the data dictionary located at the tail of the preset buffer queue, that is, the data dictionary whose time interval between the time when the last use is finished and the current time is the smallest or the data dictionary whose use frequency is the lowest within the preset time, which achieves the effect of keeping the data dictionary that has been used recently or whose use frequency is higher within the preset time in the preset buffer queue, so that, when the data dictionary of the target data table corresponding to the target data to be synchronized is kept in the preset buffer queue, the data dictionary does not need to be obtained from the source database again, the number of interactions with the data synchronization software and the source database can be reduced, and the efficiency of data synchronization is improved.
Fig. 5 is a flowchart of steps provided by another embodiment of the present disclosure, where S1401-S1402 in fig. 5 are descriptions of an implementation manner of step S140 based on the embodiment shown in fig. 1, and referring to fig. 5, step S140 (synchronizing at least one piece of target data in a target data table to a target database by parsing a data dictionary of the target data table stored at a head of a preset buffer queue) may include:
and S1401, analyzing the data dictionary stored at the head of the preset buffer queue, and acquiring data operation corresponding to at least one piece of target data.
For example, in the case that the target data is incremental data, the data operation corresponding to at least one piece of target data may include an addition operation, a change operation, a deletion operation, and the like of the data.
S1402, synchronizing the at least one piece of target data to the target database based on the data operation corresponding to the at least one piece of target data.
Optionally, before synchronizing the at least one piece of target data to the target database, synchronizing the at least one piece of target data to the target database further includes determining a synchronization data table corresponding to the target data table in the target database. Determining the implementation manner of the synchronous data table corresponding to the target data table in the target database may include: analyzing a data dictionary of the target data table, acquiring a table name of the target data table, and determining a synchronous data table corresponding to the target data table in the target database based on the table name of the target data table, wherein the table name of the target data table is the same as the table name of the synchronous data table corresponding to the target data table in the target database. And further synchronizing at least one piece of target data to a synchronous data table corresponding to the target data table.
Optionally, an implementation manner of synchronizing the at least one piece of target data to the target database based on the data operation corresponding to the at least one piece of target data includes: constructing a synchronous statement based on data operation corresponding to at least one piece of target data; and synchronizing the at least one piece of target data to the target database by executing the synchronization statement.
A way to construct a synchronization sentence may include: the transaction information of the target data is obtained through the log file of the source database and the data dictionary of the target data table, the data operation corresponding to at least one piece of target data is determined, and a corresponding synchronous statement is constructed according to the data operation corresponding to at least one piece of target data, wherein the synchronous statement can be an SQL statement. Illustratively, if the operation on the target data in the target data table is an Insert operation, an Insert (Insert) statement is constructed; if the operation on the target data in the target data table is an updating operation, constructing an updating (Update) statement; if the operation on the target data in the target data table is a Delete operation, a Delete (Delete) statement is constructed.
Based on the above-constructed synchronization statement, synchronizing the at least one piece of target data to the target database may include: if the operation on the target data in the target data table is an inserting operation, inserting the target data into the corresponding synchronous data table based on the constructed inserting statement; if the operation on the target data in the target data table is an updating operation, updating the target data of the corresponding synchronous data table based on the constructed updating statement; and if the operation on the target data in the target data table is a deletion operation, deleting the target data in the corresponding synchronous data table based on the constructed deletion statement.
Based on the same inventive concept, as an implementation of the foregoing method, an embodiment of the present invention further provides a device for executing the data processing method provided in the foregoing embodiment, where the device may execute the data processing method corresponding to the foregoing method embodiment, and for convenience of reading, details in the foregoing method embodiment are not repeated one by one in this device embodiment, but it should be clear that the data processing device in this embodiment can correspondingly implement all the contents in the foregoing method embodiment.
Fig. 6 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present disclosure, and as shown in fig. 6, a data processing apparatus 600 according to the embodiment includes:
an obtaining module 610, configured to obtain a data dictionary of a target data table in a database, where the target data table includes at least one piece of target data;
the determining module 620 is configured to determine whether the number of the data dictionaries stored in the preset buffer queue at the current time is smaller than a first threshold;
the processing module 630 is configured to delete the data dictionary stored at the tail of the preset cache queue at the current time from the preset cache queue and store the data dictionary of the target data table to the head of the preset cache queue if the number of the data dictionaries stored in the preset cache queue at the current time is not less than a first threshold;
a synchronizing module 640, configured to synchronize at least one piece of target data in the target data table to a target database by analyzing the data dictionary of the target data table stored at the head of the preset buffer queue.
As an optional implementation manner of the embodiment of the present disclosure, the determining module 620 is further configured to determine whether a data dictionary of the target data table is stored in the preset cache queue; if so, moving the data dictionary of the target data table to the head of the preset cache queue; and if not, acquiring a data dictionary of the target data table from the source database.
As an optional implementation manner of the embodiment of the present disclosure, the obtaining module 610 is specifically configured to obtain a log file of a source database, and obtain an ID identifier of a data table corresponding to target data through the log file of the source database to obtain the ID identifier of the target data table; and acquiring a data dictionary of the target data table from the source database according to the ID of the target data table.
Fig. 7 is a schematic structural diagram of a data processing apparatus according to another embodiment of the present disclosure, and based on the data processing apparatus shown in fig. 6, the data processing apparatus 600 may further include: the analysis module 650 is configured to, if the number of the data dictionaries stored in the preset cache queue at the current time is smaller than a first threshold, store the data dictionary of the target data table to the head of the preset cache queue, and not delete the data dictionary stored at the tail of the preset cache queue at the current time.
As an optional implementation manner of the embodiment of the present disclosure, the synchronization module 640 includes an analysis unit and an execution unit; the analysis unit is used for analyzing the data dictionary stored at the head of the preset cache queue to obtain data operation corresponding to the at least one piece of target data; the execution unit is used for synchronizing the at least one piece of target data to a target database based on the data operation corresponding to the at least one piece of target data.
As an optional implementation manner of the embodiment of the present disclosure, the execution unit is specifically configured to construct a synchronization statement based on a data operation corresponding to the at least one piece of target data; synchronizing the at least one piece of target data to a target database by executing the synchronization statement.
As an optional implementation manner of the embodiment of the present disclosure, the processing module 630 is specifically configured to construct a cache node corresponding to the target data table based on information included in the data dictionary of the target data table; and storing the cache node corresponding to the target data table to the head of the preset cache queue.
The data processing apparatus provided in this embodiment may execute the data processing method provided in the foregoing method embodiment, and the implementation principle and the technical effect are similar, which are not described herein again. The various modules in the data processing apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, an electronic device is provided, comprising a memory storing a computer program and a processor implementing the steps of any of the data processing methods described in the above method embodiments when the computer program is executed by the processor.
Exemplarily, fig. 8 is a schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. As shown in fig. 8, the electronic device provided in this embodiment includes: a memory 81 and a processor 82, the memory 81 being for storing computer programs; the processor 82 is configured to execute the steps in the data processing method provided by the foregoing method embodiment when calling the computer program, and the implementation principle and the technical effect are similar, which are not described herein again. Those skilled in the art will appreciate that the architecture shown in fig. 8 is merely a block diagram of some of the structures associated with the disclosed aspects and does not constitute a limitation on the electronic devices to which the disclosed aspects apply, as a particular electronic device may include more or less components than those shown, or combine certain components, or have a different arrangement of components.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, carries out the steps of any one of the data processing methods described above in the method embodiments.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, the computer program can include the processes of the embodiments of the methods described above. Any reference to memory, databases, or other media used in the embodiments provided by the present disclosure may include at least one of non-volatile and volatile memory. Non-volatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical storage, or the like. Volatile Memory can include Random Access Memory (RAM) or external cache Memory. By way of illustration and not limitation, RAM is available in many forms, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), and the like.
It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (10)
1. A data processing method, comprising:
acquiring a data dictionary of a target data table from a source database, wherein the target data table comprises at least one piece of target data;
judging whether the number of the data dictionaries stored in the preset cache queue at the current moment is less than a first threshold value or not;
if not, deleting the data dictionary stored at the tail of the preset cache queue at the current moment from the preset cache queue, and storing the data dictionary of the target data table to the head of the preset cache queue;
and synchronizing at least one piece of target data in the target data table to a target database by analyzing the data dictionary of the target data table stored at the head of the preset cache queue.
2. The method of claim 1, wherein prior to obtaining the data dictionary for the target data table in the source database, the method further comprises:
judging whether a data dictionary of the target data table is stored in the preset cache queue or not;
if so, moving the data dictionary of the target data table to the head of the preset cache queue;
and if not, acquiring a data dictionary of the target data table from the source database.
3. The method of claim 1, wherein obtaining the data dictionary of the target data table in the source database comprises:
acquiring a log file of a source database in the source database, and acquiring an ID (identity) of a data table corresponding to target data through the log file of the source database to obtain an ID of the target data table;
and acquiring a data dictionary of the target data table from the source database according to the ID of the target data table.
4. The method of claim 1, further comprising:
and if the number of the data dictionaries stored in the preset cache queue at the current moment is less than a first threshold value, storing the data dictionary of the target data table to the head of the preset cache queue, and not deleting the data dictionary stored at the tail of the preset cache queue at the current moment.
5. The method of claim 1, wherein the synchronizing at least one piece of target data in the target data table to a target database by parsing the data dictionary stored at the head of the preset buffer queue comprises:
analyzing the data dictionary stored at the head of the preset cache queue to obtain data operation corresponding to the at least one piece of target data;
and synchronizing the at least one piece of target data to a target database based on the data operation corresponding to the at least one piece of target data.
6. The method of claim 5, wherein the synchronizing the at least one target datum to a target database based on the data operation corresponding to the at least one target datum comprises:
constructing a synchronous statement based on the data operation corresponding to the at least one piece of target data;
synchronizing the at least one piece of target data to a target database by executing the synchronization statement.
7. The method of claim 1, wherein the saving the data dictionary of the target data table to the head of the preset buffer queue comprises:
constructing a cache node corresponding to the target data table based on information included in the data dictionary of the target data table;
and storing the cache node corresponding to the target data table to the head of the preset cache queue.
8. A data processing apparatus, comprising:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring a data dictionary of a target data table in a database, and the target data table comprises at least one piece of target data;
the judging module is used for judging whether the number of the data dictionaries stored in the preset cache queue at the current moment is less than a first threshold value or not;
the processing module is used for deleting the data dictionary stored at the tail of the preset cache queue at the current moment from the preset cache queue and storing the data dictionary of the target data table to the head of the preset cache queue if the number of the data dictionaries stored in the preset cache queue at the current moment is not less than a first threshold value;
and the synchronization module is used for synchronizing at least one piece of target data in the target data table to a target database by analyzing the data dictionary of the target data table stored at the head of the preset cache queue.
9. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the data processing method of any one of claims 1-7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111650609.XA CN114328738A (en) | 2021-12-30 | 2021-12-30 | Data processing method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111650609.XA CN114328738A (en) | 2021-12-30 | 2021-12-30 | Data processing method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114328738A true CN114328738A (en) | 2022-04-12 |
Family
ID=81019213
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111650609.XA Pending CN114328738A (en) | 2021-12-30 | 2021-12-30 | Data processing method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114328738A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117896440A (en) * | 2024-03-15 | 2024-04-16 | 江西曼荼罗软件有限公司 | Data caching acquisition method and system |
| CN117896440B (en) * | 2024-03-15 | 2024-05-24 | 江西曼荼罗软件有限公司 | Data caching acquisition method and system |
-
2021
- 2021-12-30 CN CN202111650609.XA patent/CN114328738A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117896440A (en) * | 2024-03-15 | 2024-04-16 | 江西曼荼罗软件有限公司 | Data caching acquisition method and system |
| CN117896440B (en) * | 2024-03-15 | 2024-05-24 | 江西曼荼罗软件有限公司 | Data caching acquisition method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11468093B2 (en) | Synopsis based advanced partition elimination | |
| US11734258B2 (en) | Constraint data statistics | |
| CN107665255B (en) | Method, device, equipment and storage medium for key value database data change | |
| CN105373541A (en) | Processing method and system for data operation request of database | |
| US11216412B2 (en) | Intelligent merging for efficient updates in columnar databases | |
| CN111414403A (en) | Data access method and device and data storage method and device | |
| US11640411B2 (en) | Data replication system | |
| CN112579602A (en) | Multi-version data storage method and device, computer equipment and storage medium | |
| CN110569255A (en) | Paging query method and device for database sub-tables and computer equipment | |
| CN112612799A (en) | Data synchronization method and terminal | |
| CN111488736B (en) | Self-learning word segmentation method, device, computer equipment and storage medium | |
| CN111046106A (en) | Cache data synchronization method, device, equipment and medium | |
| EP3267329A1 (en) | Data processing method having structure of cache index specified to transaction in mobile environment dbms | |
| CN114328738A (en) | Data processing method and device, electronic equipment and readable storage medium | |
| KR100624863B1 (en) | Method for effectively using memory in mobile station | |
| CN115469810A (en) | Data acquisition method, device, equipment and storage medium | |
| CN115098503A (en) | Null value data processing method and device, computer equipment and storage medium | |
| CN112800123A (en) | Data processing method, data processing device, computer equipment and storage medium | |
| CN113704267A (en) | Data query method, system, equipment and storage medium based on elastic search | |
| CN113296693A (en) | Data storage method and device | |
| CN107436918B (en) | Database implementation method, device and equipment | |
| US10795875B2 (en) | Data storing method using multi-version based data structure | |
| CN112527911B (en) | Data storage method, device, equipment and medium | |
| CN114385753A (en) | Database data synchronization method and device based on data page preloading | |
| CN116450751A (en) | Transaction processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |