CN115525666A - Real-time data updating method and device, electronic equipment and storage medium - Google Patents

Real-time data updating method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN115525666A
CN115525666A CN202211209286.5A CN202211209286A CN115525666A CN 115525666 A CN115525666 A CN 115525666A CN 202211209286 A CN202211209286 A CN 202211209286A CN 115525666 A CN115525666 A CN 115525666A
Authority
CN
China
Prior art keywords
data
transaction
real
transaction number
local
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211209286.5A
Other languages
Chinese (zh)
Inventor
古毅伟
竹记
林雯
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Pudong Development Bank Co Ltd
Original Assignee
Shanghai Pudong Development Bank Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Pudong Development Bank Co Ltd filed Critical Shanghai Pudong Development Bank Co Ltd
Priority to CN202211209286.5A priority Critical patent/CN115525666A/en
Publication of CN115525666A publication Critical patent/CN115525666A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • G06F16/2379Updates performed during online database operations; commit processing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/466Transaction processing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/546Message passing systems or structures, e.g. queues
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00Indexing scheme relating to G06F9/00
    • G06F2209/54Indexing scheme relating to G06F9/54
    • G06F2209/548Queue

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention discloses a real-time data updating method, a real-time data updating device, electronic equipment and a storage medium, wherein the real-time data updating method applied to a data processing server comprises the following steps: polling the target queue according to the data mark of the cache queue to sequentially acquire distributed transaction processing data of each local transaction number under the global transaction number corresponding to the real-time data updating operation; the target queue comprises a message queue and/or a buffer queue; and synchronously updating the distributed transaction processing data acquired in sequence to an aggregation database according to the transaction processing sequence. The technical scheme of the embodiment of the invention can keep the time sequence of real-time data updating, thereby improving the accuracy and reliability of real-time data updating.

Description

Real-time data updating method and device, electronic equipment and storage medium
Technical Field
The embodiment of the invention relates to the technical field of data processing, in particular to a real-time data updating method and device, electronic equipment and a storage medium.
Background
With the high-speed development of internet application, the characteristics of massive data, large flow, high concurrency and the like of the internet application are more prominent and obvious, and the traditional database architecture in a master-slave mode cannot meet the high-speed development of internet application services. In order to improve the performance of the system, a data slicing architecture is adopted in many IT (Internet Technology) system architecture designs. The data slicing is to slice data according to a slicing key (such as a client number or an organization number in the financial field), and store the data of different slices independently, for example, each slice data is stored in a database.
In the data processing architecture, when a data query request is processed, data query which can belong to the same fragment is very efficient, but joint query across data fragments is difficult to realize, and query efficiency is low. In the related technology, in order to improve the efficiency of data query across data shards, the proposed solution is to convert all shard database data and then synchronize the processed data to an aggregated database for table-linked query, where the aggregated database may be implemented by using a large data storage technology.
In the process of implementing the invention, the inventor finds that the prior art has the following defects: in the process of data synchronization updating of the aggregated database, since the aggregated database is used for synchronizing data updating of different source fragment data, even if the aggregated database has a global transaction control function on the source fragment database, the data updating cannot be performed according to a local transaction submission sequence of the source fragment database. That is, the aggregated database cannot maintain the ordering of the source database updates.
Disclosure of Invention
Embodiments of the present invention provide a real-time data updating method and apparatus, an electronic device, and a storage medium, which can maintain the time sequence of real-time data updating, thereby improving the accuracy and reliability of real-time data updating.
According to an aspect of the present invention, there is provided a real-time data updating method applied to a data processing server, including:
polling the target queue according to the cache queue data mark to sequentially acquire distributed transaction processing data of each local transaction number under the global transaction number corresponding to the real-time data updating operation; the target queue comprises a message queue and/or a buffer queue;
and synchronously updating the distributed transaction processing data acquired in sequence to an aggregation database according to the transaction processing sequence.
According to another aspect of the present invention, there is provided a real-time data updating method applied to a local transaction processor, including:
acquiring a local transaction number issued by the global transaction processor according to the real-time data updating operation;
executing the distributed transaction according to the local transaction number to obtain distributed transaction processing data;
pushing the distributed transaction processing data to a message queue for storage;
and the distributed transaction processing data is used for synchronously updating the data processing server to the aggregation server according to the transaction processing sequence.
According to another aspect of the present invention, there is provided a real-time data updating method applied to a global transaction processor, including:
responding to a real-time data updating operation, and generating a global transaction number and a local transaction number corresponding to the real-time data updating operation;
sending the global transaction number and the local transaction number corresponding to the real-time data updating operation to a data processing server;
receiving transaction number association information fed back by the data processing server, and pushing the local transaction number to a local transaction processor under the condition that the transaction number association information is verified, so that the local transaction processor executes a distributed transaction according to the local transaction number to obtain distributed transaction processing data;
and the distributed transaction processing data is used for synchronously updating the data processing server to the aggregation server according to the transaction processing sequence.
According to another aspect of the present invention, there is provided a real-time data updating method, including:
responding to a real-time data updating operation through a global transaction processor, generating a global transaction number and a local transaction number corresponding to the real-time data updating operation, and sending the global transaction number and the local transaction number corresponding to the real-time data updating operation to a data processing server; receiving the transaction number association information fed back by the data processing server, and pushing the local transaction number to a local transaction processor under the condition that the transaction number association information is verified;
acquiring a local transaction number issued by a global transaction processor according to real-time data updating operation through a local transaction processor, executing a distributed transaction according to the local transaction number to obtain distributed transaction processing data, and pushing the distributed transaction processing data to a message queue for storage;
polling the target queue through the data processing server according to the data mark of the cache queue to sequentially obtain distributed transaction processing data of each local transaction number under the global transaction number corresponding to the real-time data updating operation; the target queue comprises a message queue and/or a buffer queue; and synchronously updating the distributed transaction processing data acquired in sequence to an aggregation database according to the transaction processing sequence.
According to another aspect of the present invention, there is provided a real-time data updating system, including a global transaction processor, a local transaction processor and a data processing server, the global transaction processor being communicatively connected to the local transaction processor, the global transaction processor being further communicatively connected to the data processing server, wherein:
the global transaction processor is to: responding to a real-time data updating operation, generating a global transaction number and a local transaction number corresponding to the real-time data updating operation, and sending the global transaction number and the local transaction number corresponding to the real-time data updating operation to a data processing server; receiving the transaction number association information fed back by the data processing server, and pushing the local transaction number to a local transaction processor under the condition that the transaction number association information is verified;
the local transaction processor is to: acquiring a local transaction number issued by a global transaction processor according to real-time data updating operation, executing a distributed transaction according to the local transaction number to obtain distributed transaction processing data, and pushing the distributed transaction processing data to a message queue for storage;
the data processing server is configured to: polling the target queue according to the data mark of the cache queue to sequentially acquire distributed transaction processing data of each local transaction number under the global transaction number corresponding to the real-time data updating operation; the target queue comprises a message queue and/or a buffer queue; and synchronously updating the distributed transaction processing data acquired in sequence to an aggregation database according to the transaction processing sequence.
According to another aspect of the present invention, there is provided a real-time data update apparatus configured at a data processing server, including:
the distributed transaction processing data acquisition module is used for polling the target queue according to the data mark of the cache queue so as to sequentially acquire distributed transaction processing data of each local transaction number under the global transaction number corresponding to the real-time data updating operation; the target queue comprises a message queue and/or a buffer queue;
and the distributed transaction processing data updating module is used for synchronously updating the distributed transaction processing data acquired in sequence to the aggregation database according to the transaction processing sequence.
According to another aspect of the present invention, there is provided a real-time data updating apparatus disposed in a local transaction processor, including:
the local transaction number acquisition module is used for acquiring a local transaction number issued by the global transaction processor according to the real-time data updating operation;
the distributed transaction processing data generation module is used for executing the distributed transaction according to the local transaction number to obtain distributed transaction processing data;
the distributed transaction processing data pushing module is used for pushing the distributed transaction processing data to a message queue for storage;
and the distributed transaction processing data is used for synchronously updating the data processing server to the aggregation server according to the transaction processing sequence.
According to another aspect of the present invention, there is provided a real-time data updating apparatus configured in a global transaction processor, including:
the transaction number generation module is used for responding to a real-time data updating operation and generating a global transaction number and a local transaction number corresponding to the real-time data updating operation;
the transaction number sending module is used for sending the global transaction number and the local transaction number corresponding to the real-time data updating operation to a data processing server;
the local transaction number pushing module is used for receiving the transaction number associated information fed back by the data processing server, and pushing the local transaction number to a local transaction processor under the condition that the transaction number associated information is verified, so that the local transaction processor executes a distributed transaction according to the local transaction number to obtain distributed transaction processing data;
and the distributed transaction processing data is used for synchronously updating the data processing server to the aggregation server according to the transaction processing sequence.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the real-time data update method according to any of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the real-time data updating method according to any one of the embodiments of the present invention when the computer instructions are executed.
According to the embodiment of the invention, the distributed transaction data of each local transaction number under the global transaction number corresponding to the real-time data updating operation is sequentially acquired by polling the message queue and/or the cache queue through the data processing server according to the cache queue data mark, and the sequentially acquired distributed transaction data is synchronously updated to the aggregation database according to the transaction processing sequence, so that the problem that the time sequence of real-time data updating cannot be ensured in the existing real-time data updating method is solved, the time sequence of real-time data updating can be ensured, and the accuracy and reliability of real-time data updating are improved.
It should be understood that the statements in this section are not intended to identify key or critical features of the embodiments of the present invention, nor are they intended to limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart of a real-time data updating method according to an embodiment of the present invention;
fig. 2 is a flowchart of a real-time data updating method according to a second embodiment of the present invention;
fig. 3 is a flowchart of a real-time data updating method according to a third embodiment of the present invention;
fig. 4 is a flowchart of a real-time data updating method according to a fourth embodiment of the present invention;
fig. 5 is a schematic diagram of a real-time data updating system according to a fifth embodiment of the present invention;
fig. 6 is a schematic diagram of a real-time data updating apparatus according to a sixth embodiment of the present invention;
fig. 7 is a schematic diagram of a real-time data updating apparatus according to a seventh embodiment of the present invention;
fig. 8 is a schematic diagram of a real-time data updating apparatus according to an eighth embodiment of the present invention;
fig. 9 is a schematic structural diagram of an electronic device according to a ninth embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example one
Fig. 1 is a flowchart of a real-time data updating method according to an embodiment of the present invention, where this embodiment is applicable to a case where distributed transaction data is updated synchronously in a chronological order through a data processing server, a message queue and a cache queue, and the method may be executed by a real-time data updating apparatus configured in the data processing server, where the apparatus may be implemented by software and/or hardware, and may be generally integrated in an electronic device, where the electronic device may be a server device, and is used in cooperation with a global transaction processor and a local transaction processor. Accordingly, as shown in fig. 1, the method comprises the following operations:
s110, polling the target queue according to the data mark of the cache queue to sequentially obtain distributed transaction processing data of each local transaction number under the global transaction number corresponding to the real-time data updating operation; wherein the target queue comprises a message queue and/or a buffer queue.
The data processing server may be used as a data aggregator to consume data in the message queue, implement aggregation operation on data submitted in the distributed Transaction, and may establish a communication connection with a Global Transaction Service (GTS) to perform interaction. The buffer queue data flag may be used to identify whether buffered data is present in the buffer queue. The real-time data update operation may be a business operation performed on the data in real-time. Alternatively, the user may perform a business operation by executing a business operation at the system front end of the real-time data update system, which may cause real-time update of the data stored in the database, and thus may act as a real-time data update operation. The global transaction number may be one transaction number generated each time the front-end system initiates a complete real-time data update operation. That is, one global transaction number may correspond to one real-time data update operation. The local transaction number may be a number corresponding to each distributed transaction data. It should be noted that one global transaction number may correspondingly include a plurality of local transaction numbers, and each local transaction number included in one global transaction number has a temporal order. The distributed transaction data may be data resulting from processing of a distributed transaction by a local transaction processor. The distributed transaction may refer to that a participant of the transaction, a server supporting the transaction, a resource server, and a transaction manager are respectively located on different node devices of different distributed systems, and may also be referred to as a local transaction. The local transaction processor may then be a processor running in a distributed database. The message queue may be responsible for receiving distributed transaction data submitted by the local transaction processor after completing the distributed transaction for consumption by the data processing server. The cache queue may be responsible for caching the distributed transaction data obtained from the message queue for subsequent use by the data processing server.
In the embodiment of the present invention, when the data processing server determines that the distributed transaction data of the real-time data update operation needs to be updated in real time, the distributed transaction data of each local transaction number under the global transaction number corresponding to the real-time data update operation may be sequentially obtained by polling the message queue and/or the cache queue. The distributed transaction processing data of each local transaction number under the global transaction number corresponding to the real-time data update operation may be obtained sequentially according to a time sequence between the distributed transaction processing data.
In an optional embodiment of the present invention, the polling the target queue according to the data flag of the cache queue to sequentially obtain the distributed transaction data of each local transaction number under the global transaction number corresponding to the real-time data update operation may include: polling the message queue to obtain distributed transaction processing data of the current target local transaction number according to the transaction processing sequence under the condition that the mark data of the data mark of the cache queue is determined to be first mark data; polling the message queue and the cache queue to obtain distributed transaction processing data of the current target local transaction number according to the transaction processing sequence under the condition that the mark data of the cache queue data mark is determined to be second mark data; and when the distributed transaction processing data of the current local transaction number acquired by polling the target queue is not the distributed transaction processing data of the current target local transaction number, the distributed transaction processing data of the current local transaction number is used for caching into the cache queue.
The first flag data may be used to identify data that is not buffered in the buffer queue, for example, the first flag data may take a false value. Accordingly, the second flag data may be used to identify that there is data buffered in the buffer queue, for example, the second flag data may take the value of true. Optionally, the flag data of the buffer queue data flag of the buffer queue may be the first flag data by default. The current target local transaction number may be a local transaction number currently required to be processed as determined by the chronological order of the local transaction numbers. For example, assume that each local transaction number is specifically 001, 002, 003. If the data processing server processes the distributed transaction data for the first time, the current target local transaction number may be 001, that is, the distributed transaction data with the number 001 needs to be updated in real time. After the data processor completes the update operation on the distributed transaction data with the number 001, the current target local transaction number is updated to be 002, that is, the distributed transaction data with the number 002 needs to be updated in real time. The transaction processing sequence is also the time sequence for processing each local transaction number.
Specifically, when the data processing server polls the target queue for the first time, since the default value of the tag data of the data tag of the cache queue is the first tag data, and at this time, the data processing server may poll the message queue to obtain the distributed transaction data of the current target local transaction number, that is, the distributed transaction data of the first local transaction number, from the message queue according to the transaction sequence. In the subsequent data processing process, if the tag data of the buffer queue data tag of the buffer queue is the second tag data, and at this time, the buffer queue has buffer data, the data processing server may poll the message queue and the buffer queue, so as to obtain the distributed transaction data of the current target local transaction number from the message queue or the buffer queue according to the transaction processing sequence.
Correspondingly, in the process of polling the message queue, if the data currently acquired by the data processing server from the message queue is not the distributed transaction processing data of the current target local transaction number, the data processing server may cache the currently acquired data in the cache queue. At this time, if the flag data of the buffer queue data flag is the first flag data, it needs to be set as the second flag data. If the data currently acquired by the data processing server from the message queue is the distributed transaction data of the current target local transaction number, the data processing server may continue to poll the target queue, or synchronously update the acquired distributed transaction data of the current target local transaction number to the database.
S120, synchronously updating the distributed transaction processing data acquired in sequence to an aggregation database according to the transaction processing sequence.
Optionally, the sequentially acquired distributed transaction data is synchronously updated to the aggregation database according to the transaction sequence, where each time one distributed transaction data is sequentially acquired, that is, the distributed transaction data is synchronously updated to the aggregation database; or after all distributed transaction data corresponding to the real-time data update operation are acquired according to the transaction processing sequence, all distributed transaction data are synchronously updated to the aggregation database according to the transaction processing sequence, as long as the aggregation database can be updated according to the time sequence of the distributed transaction data, and the embodiment of the invention does not limit the update mode of the distributed transaction data.
In an optional embodiment of the present invention, after synchronously updating the sequentially acquired distributed transaction data to the aggregation database according to the transaction order, the method may further include: resetting the flag data of the buffer queue data flag to the first flag data.
In a specific example, the data processing server obtains the local transaction numbers LT001, LT002 and LT003 corresponding to the global transaction number GT001 from the global transaction processor, and LT001, LT002 and LT003 may form a group of transaction number groups. Further, the data processing server polls the distributed transaction processing data corresponding to the local transaction numbers from the message queue and/or the cache queue, and aggregates the polled distributed transaction processing data into the aggregation database.
Step one, the data processing server polls the message queue, and if the data processing server acquires the first local transaction number in the sequence of the transaction number groups LT001, LT002 and LT003, namely the distributed transaction processing data corresponding to LT001, the first local transaction number is synchronously updated to the aggregated data. If the transaction data is not the distributed transaction data corresponding to the LT001, the data processing server stores the transaction data into the cache queue, and sets a cache queue data flag associated with the global transaction number GT001 as true in the cache queue. Or, the data processing server may also cache the acquired distributed transaction data corresponding to LT001, for example, cache the distributed transaction data to the local, and wait to acquire distributed transaction data of other local transaction numbers in sequence.
And secondly, if the data processing server completes processing of the distributed transaction data corresponding to the LT001 and the data processing server detects that the cache queue data mark associated with the global transaction number GT001 in the cache queue is true, polling data from the message queue and the cache queue respectively. And if the data processing server acquires the distributed transaction processing data corresponding to the second local transaction number, namely LT002, in the sequence of the transaction number groups LT001, LT002 and LT003, synchronously updating the distributed transaction processing data to the aggregated data. If the transaction data is not the distributed transaction data corresponding to the LT002, the data processing server saves the transaction data into the cache queue. At this time, if the cache queue data flag associated with the global transaction number GT001 is set as true in the cache queue, the cache queue data flag does not need to be modified, and if the cache queue data flag associated with the global transaction number GT001 is set as false in the cache queue, the cache queue data flag associated with the global transaction number GT001 needs to be set as true. If the data processing server completes processing of the distributed transaction processing data corresponding to the LT001 and the data processing server detects that the cache queue data flag associated with the global transaction number GT001 in the cache queue is false, continuing polling data from the message queue to acquire the distributed transaction processing data corresponding to the LT 002. Or, the data processing server may also cache the acquired distributed transaction data corresponding to LT002 in the distributed transaction data location corresponding to LT001, and wait to sequentially acquire distributed transaction data of other local transaction numbers.
And step three, if the data processing server completes processing of distributed transaction processing data corresponding to LT001 and LT002 and the data processing server detects that a cache queue data mark associated with the global transaction number GT001 in the cache queue is true, polling data from the message queue and the cache queue respectively to obtain the distributed transaction processing data corresponding to LT003. If the data processing server completes processing of distributed transaction processing data corresponding to LT001 and LT002 and the data processing server detects that the cache queue data flag associated with the global transaction number GT001 in the cache queue is false, the data processing server continues polling data from the message queue to acquire distributed transaction processing data corresponding to LT003. After the data processing server acquires the distributed transaction data corresponding to LT003, it may synchronously update the distributed transaction data to the aggregated data. Or, the data processing server may cache the acquired distributed transaction data corresponding to LT003 in the distributed transaction data location corresponding to LT 002. At this time, the data cached by the data processing server in the time sequence of the local transaction number is distributed transaction data corresponding to LT001, LT002, and LT003, respectively. The data processing server can synchronously update distributed transaction data corresponding to LT001, LT002 and LT003 to the aggregation database according to the sequence of LT001-LT002-LT 003.
After the data processing server completes the update operation of the distributed transaction processing data corresponding to all the local transaction numbers LT001, LT002, and LT003 in the transaction number group corresponding to the global transaction number GT001, the cache queue data flag associated with the global transaction number in the cache queue may be reset to false.
Therefore, the distributed transaction processing data in the application data table in the distributed database are sequentially aggregated to the aggregated database used for specific query, so that the time sequence of real-time data updating in the aggregated database can be ensured, and the accuracy and reliability of real-time data updating in the aggregated database are improved.
According to the embodiment of the invention, the distributed transaction data of each local transaction number under the global transaction number corresponding to the real-time data updating operation is sequentially acquired by polling the message queue and/or the cache queue through the data processing server according to the cache queue data mark, and the sequentially acquired distributed transaction data is synchronously updated to the aggregation database according to the transaction processing sequence, so that the problem that the time sequence of real-time data updating cannot be ensured in the existing real-time data updating method is solved, the time sequence of real-time data updating can be ensured, and the accuracy and reliability of real-time data updating are improved.
Example two
Fig. 2 is a flowchart of a real-time data updating method according to a second embodiment of the present invention, where this embodiment is applicable to a case where distributed transaction data that needs to be updated in real time is generated by a local transaction processor, so as to further update the real-time data synchronously according to a chronological order through a data processing server, a message queue, and a cache queue, and the method may be executed by a real-time data updating apparatus configured in the local transaction processor, where the apparatus may be implemented by software and/or hardware, and may be generally integrated in an electronic device, where the electronic device may be a terminal device or a server device, and is used in cooperation with a global transaction processor and a data processing server. Accordingly, as shown in fig. 2, the method includes the following operations:
and S210, acquiring a local transaction number issued by the global transaction processor according to the real-time data updating operation.
The local transaction processor can be used as a transaction initiator and is responsible for starting, submitting or rolling back distributed transactions, and can establish communication connection with the global transaction processor for interaction. The global transaction processor can be used as a transaction coordinator and is responsible for promoting the distributed transaction and managing the life cycle of the transaction. Alternatively, the global transaction processor may be a processor running in a centralized database.
And S220, executing the distributed transaction according to the local transaction number to obtain distributed transaction processing data.
In the embodiment of the present invention, the global transaction processor may generate a global transaction number corresponding to the real-time data update operation and each local transaction number under the global transaction number. After the global transaction processor generates the local transaction number, the local transaction number can be issued to the local transaction processor, and the local transaction processor processes each distributed transaction of the real-time data updating operation according to the local transaction number to generate distributed transaction processing data. Optionally, there may be one or multiple local transaction processors for processing the distributed transaction corresponding to each local transaction number, which is not limited in this embodiment of the present invention.
S230, pushing the distributed transaction processing data to a message queue for storage.
And the distributed transaction processing data is used for synchronously updating the data processing server to the aggregation server according to the transaction processing sequence.
Correspondingly, after the local transaction processor obtains the distributed transaction processing data, the distributed transaction processing data can be pushed to a message queue for storage so as to be consumed by the data processing server. The data processing server may poll the message queue, and cache the distributed transaction data of the current local transaction number in the cache queue when it is determined that the acquired distributed transaction data of the current local transaction number is not the distributed transaction data of the current target local transaction number. In the subsequent process, the data processing server may poll the message queue and/or the cache queue according to the cache queue data flag, sequentially obtain the distributed transaction data of each local transaction number under the global transaction number corresponding to the real-time data updating operation, and synchronously update the sequentially obtained distributed transaction data to the aggregation database according to the transaction sequence.
According to the embodiment of the invention, the local transaction number issued by the global transaction processor according to the real-time data updating operation is acquired through the local transaction processor, the distributed transaction is executed according to the local transaction number, the distributed transaction data is obtained, and the distributed transaction data is pushed to the message queue for storage, so that the distributed transaction data is synchronously updated to the aggregation server by the data processing server according to the transaction sequence, the problem that the time sequence of real-time data updating cannot be ensured in the existing real-time data updating method is solved, the time sequence of real-time data updating can be ensured, and the accuracy and reliability of real-time data updating are improved.
EXAMPLE III
Fig. 3 is a flowchart of a real-time data updating method according to a third embodiment of the present invention, where this embodiment is applicable to a situation where a global transaction processor generates a transaction number and sends the transaction number to a local transaction processor and a data processing server, so that the local transaction processor further generates distributed transaction data that needs to be updated in real time, and the distributed transaction data is synchronously updated according to a time sequence through the data processing server, a message queue and a cache queue. Accordingly, as shown in fig. 3, the method includes the following operations:
s310, responding to the real-time data updating operation, and generating a global transaction number and a local transaction number corresponding to the real-time data updating operation.
In the embodiment of the present invention, after the user executes the service operation through the system front end of the real-time data updating system, the global transaction processor in the real-time data updating system may generate the global transaction number and the local transaction number corresponding to the real-time data updating operation in response to the real-time data updating operation.
And S320, sending the global transaction number and the local transaction number corresponding to the real-time data updating operation to a data processing server.
S330, receiving the transaction number related information fed back by the data processing server, and pushing the local transaction number to a local transaction processor under the condition that the transaction number related information is verified, so that the local transaction processor executes a distributed transaction according to the local transaction number to obtain distributed transaction processing data.
And the distributed transaction processing data is used for synchronously updating the data processing server to the aggregation server according to the transaction processing sequence. The transaction number related information may be information related to the transaction number, for example, the transaction number itself, or character information generated by the transaction number, and the like.
Further, the global transaction processor may send the generated global transaction number and the local transaction number to the data processing server for verification. After receiving the global transaction number and the local transaction number, the data processing server may feed back the transaction number association information to the global transaction processor, for example, directly feed back the received global transaction number and the local transaction number, or encrypt the global transaction number and the local transaction number by using an encryption algorithm to generate an encrypted character string, which is fed back to the global transaction processor as the transaction number association information. And after receiving the transaction number association information fed back by the data processing server, the global transaction processor verifies the transaction number association information. For example, if the transaction number association information is a global transaction number and a local transaction number, it may be directly checked whether the global transaction number and the local transaction number are the same as the transaction number information issued to the data processing server. If the transaction number association information is an encrypted character string generated by encrypting the global transaction number and the local transaction number by using an encryption algorithm, the encrypted character string can be decrypted to obtain the global transaction number and the local transaction number, and whether the global transaction number and the local transaction number are the same as the transaction number information issued to the data processing server or not is verified. The global transaction processor may push the local transaction number to the local transaction processor if it is determined that the transaction number association information passes the check. And the local transaction processor executes the distributed transaction according to the received local transaction number to obtain distributed transaction processing data. The data processing server may poll the target queue according to the received global transaction number and the received local transaction number, determine whether the distributed transaction data of the current local transaction number is the distributed transaction data of the current target local transaction number, and sequentially obtain the distributed transaction data of each local transaction number under the global transaction number corresponding to the real-time data updating operation.
The embodiment of the invention responds to the real-time data updating operation through the global transaction processor, generates the global transaction number and the local transaction number corresponding to the real-time data updating operation, sends the global transaction number and the local transaction number to the data processing server, and receives the transaction number association information fed back by the data processing server. And under the condition that the transaction number association information is verified, pushing the local transaction number to a local transaction processor, so that the local transaction processor executes the distributed transaction according to the local transaction number to obtain distributed transaction processing data. Further, the data processing server polls the message queue and/or the cache queue according to the cache queue data mark to sequentially acquire the distributed transaction data of each local transaction number under the global transaction number corresponding to the real-time data updating operation, so that the sequentially acquired distributed transaction data is synchronously updated to the aggregated database according to the transaction processing sequence, the problem that the existing real-time data updating method cannot guarantee the time sequence of real-time data updating is solved, the time sequence of real-time data updating can be guaranteed, and the accuracy and the reliability of real-time data updating are improved.
Example four
Fig. 4 is a flowchart of a real-time data updating method according to a fourth embodiment of the present invention, and as shown in fig. 4, the method includes the following operations:
s410, responding to a real-time data updating operation through a global transaction processor, and generating a global transaction number and a local transaction number corresponding to the real-time data updating operation.
And S420, sending the global transaction number and the local transaction number corresponding to the real-time data updating operation to a data processing server through a global transaction processor.
S430, receiving the transaction number related information fed back by the data processing server through the global transaction processor, and pushing the local transaction number to the local transaction processor under the condition that the transaction number related information is verified.
S440, local transaction numbers issued by the global transaction processor according to real-time data updating operation are obtained through the local transaction processor, and distributed transactions are executed according to the local transaction numbers to obtain distributed transaction processing data.
S450, pushing the distributed transaction processing data to a message queue for storage through a local transaction processor.
And S460, polling the message queue and/or the buffer queue through the data processing server according to the buffer queue data mark to sequentially obtain the distributed transaction processing data of each local transaction number under the global transaction number corresponding to the real-time data updating operation.
S470, synchronously updating the distributed transaction processing data acquired in sequence to an aggregation database according to the transaction processing sequence through the data processing server.
Optionally, the data processing server is specifically configured to: polling the message queue to obtain distributed transaction processing data of the current target local transaction number according to the transaction processing sequence under the condition that the mark data of the data mark of the cache queue is determined to be first mark data; polling the message queue and the cache queue to obtain distributed transaction processing data of the current target local transaction number according to the transaction processing sequence under the condition that the mark data of the cache queue data mark is determined to be second mark data; and when the distributed transaction processing data of the current local transaction number acquired by polling the target queue is not the distributed transaction processing data of the current target local transaction number, the distributed transaction processing data of the current local transaction number is used for caching into the cache queue.
Optionally, the data processing server is further configured to: and resetting the mark data of the data mark of the cache queue to the first mark data after synchronously updating the distributed transaction processing data acquired in sequence to an aggregation database according to the transaction processing sequence.
In a specific example, the real-time data updating method may specifically include the following operations:
step one, prepositive preparation work:
step 1.1, each time a user executes a real-time data updating operation through the system front end of the real-time data updating system, the global transaction server generates a global transaction number such as GT001 for the real-time data updating operation, and generates a group of sequential local transaction numbers LT001, LT002 and LT003 corresponding to the global transaction number GT 001. Alternatively, a set of local transaction numbers may be collectively referred to as a transaction number set. The order of the local transaction numbers in the transaction number group represents the local transaction commit order intended by the global transaction server when committing the global transaction.
Step 1.2, the global transaction processor establishes communication connection with the data processing server, and sends the global transaction number GT001 corresponding to the business operation and the included local transaction numbers LT001, LT002 and LT003 to the data processing server. And the data processing server confirms the receipt after receiving the transaction number and returns the transaction number association information to the global transaction processor. Meanwhile, the data processing server may locally store the received transaction number.
And step 1.3, the global transaction processor receives the transaction number association information returned by the data processing server, checks the transaction number association information and sends the group of local transaction numbers to the local transaction processor after confirming no errors.
And step two, the local transaction processor performs distributed transaction according to the local transaction numbers LT001, LT002 and LT003, and submits the content to the message queue after completion.
Step 2.1, the local transaction handler performs the opening, committing and rollback of the distributed transaction according to local transaction numbers LT001, LT002 and LT003. The local transaction number LT001 may correspond to an opened distributed transaction, the local transaction number LT002 may correspond to a submitted distributed transaction, and the local transaction number LT003 may correspond to a rolled-back distributed transaction.
And 2.2, after the local transaction processor finishes processing the distributed transaction, pushing the obtained distributed transaction processing data to a message queue. It will be appreciated that the distributed transaction data stored in the message queue may be out of order or stored sequentially due to various factors such as network quality.
And step three, the data processing server polls data from the message queue and the cache queue according to the global transaction number GT001 and the corresponding transaction number groups LT001, LT002 and LT003 acquired from the global transaction processor, and synchronously updates the data to the aggregation server according to the transaction processing sequence.
And 3.1, polling the message queue by the data processing server, and if the distributed transaction processing data of the current target local transaction number in the transaction number group LT001, LT002 and LT003 sequence, such as the distributed transaction processing data of LT001, synchronously updating the distributed transaction processing data to the aggregation server. If the distributed transaction data is not the distributed transaction data of the current target local transaction number, the distributed transaction data is stored in the cache queue, and the flag bit associated with the global transaction number GT001 of the cache queue is set to true.
And 3.2, in the process of polling data in the later period, if the flag bit associated with the global transaction number GT001 in the cache queue is true, the data processing server polls the data from the message queue and the cache queue respectively. If the distributed transaction data of the current target local transaction number in the sequence of the transaction number groups LT001, LT002 and LT003, such as LT002, is acquired, the distributed transaction data is synchronously updated to the aggregation server. If the flag bit associated with the global transaction number in the buffer queue is false, the operation of step 3.1 is performed.
And 3.3, after the data processing server completes the synchronous updating operation of the distributed transaction processing data of all the local transaction numbers LT001, LT002 and LT003 in the transaction number group corresponding to the global transaction number GT001, setting the flag bit associated with the global transaction number in the cache queue as false, and disconnecting the communication connection with the global transaction processor.
It should be noted that, after polling the distributed transaction data of the current target local transaction number, the data processing server may temporarily not update the distributed transaction data to the aggregation server in synchronization, but cache the distributed transaction data locally. After the data processing server caches the distributed transaction processing data of all local transaction numbers in sequence, statistics can be carried out on the cached distributed transaction processing data, and the cached distributed transaction processing data can be synchronously updated to the aggregation database according to the transaction processing sequence.
Compared with the existing real-time data updating method, the real-time data updating method can effectively solve the problem of orderliness in the data aggregation process, and further ensure the orderliness of querying database data after aggregation. By the real-time data aggregation method for keeping the orderliness, the problem that data query and actual service operation results are different after data aggregation is carried out on fragment database data can be solved.
The embodiment of the invention responds to the real-time data updating operation through the global transaction processor, generates the global transaction number and the local transaction number corresponding to the real-time data updating operation, sends the global transaction number and the local transaction number to the data processing server, and receives the transaction number association information fed back by the data processing server. And under the condition that the transaction number association information is verified, pushing the local transaction number to a local transaction processor, so that the local transaction processor executes the distributed transaction according to the local transaction number to obtain distributed transaction processing data. Further, the data processing server polls the message queue and/or the cache queue according to the cache queue data mark, and sequentially obtains the distributed transaction data of each local transaction number under the global transaction number corresponding to the real-time data updating operation, so that the sequentially obtained distributed transaction data are synchronously updated to the aggregated database according to the transaction sequence, the problem that the existing real-time data updating method cannot guarantee the time sequence of real-time data updating is solved, the time sequence of real-time data updating can be guaranteed, and the accuracy and reliability of real-time data updating are improved.
EXAMPLE five
Fig. 5 is a schematic diagram of a real-time data updating system according to a fifth embodiment of the present invention, where the structure of the real-time data updating system includes: a global transaction processor 510, a local transaction processor 520, and a data processing server 530, the global transaction processor 510 being communicatively coupled to the local transaction processor 520, the global transaction processor 510 being further communicatively coupled to the data processing server 530, wherein:
global transaction processor 510 is used to: responding to the real-time data updating operation, generating a global transaction number and a local transaction number corresponding to the real-time data updating operation, and sending the global transaction number and the local transaction number corresponding to the real-time data updating operation to the data processing server 530; receiving the transaction number association information fed back by the data processing server 530, and pushing the local transaction number to the local transaction processor 520 when determining that the transaction number association information passes the verification;
local transaction processor 520 is configured to: acquiring a local transaction number issued by the global transaction processor 510 according to a real-time data updating operation, executing a distributed transaction according to the local transaction number to obtain distributed transaction processing data, and pushing the distributed transaction processing data to the message queue 540 for storage;
the data processing server 530 is configured to: polling the target queue according to the data mark of the cache queue to sequentially acquire distributed transaction processing data of each local transaction number under the global transaction number corresponding to the real-time data updating operation; wherein the target queue comprises a message queue 540 and/or a buffer queue 550; and synchronously updating the distributed transaction processing data acquired in sequence to an aggregation database according to the transaction processing sequence.
Optionally, the data processing server 530 is specifically configured to: polling the message queue 540 to obtain the distributed transaction processing data of the current target local transaction number according to the transaction processing sequence under the condition that the mark data of the cache queue data mark is determined to be the first mark data; polling the message queue 540 and the cache queue 550 to obtain the distributed transaction data of the current target local transaction number according to the transaction sequence under the condition that the flag data of the cache queue data flag is determined to be the second flag data; when the distributed transaction data of the current local transaction number acquired by polling the target queue is not the distributed transaction data of the current target local transaction number, the distributed transaction data of the current local transaction number is used for caching in the cache queue 550.
Optionally, the data processing server 530 is further configured to: and resetting the mark data of the data mark of the cache queue to the first mark data after synchronously updating the distributed transaction processing data acquired in sequence to an aggregation database according to the transaction processing sequence.
The embodiment of the invention responds to the real-time data updating operation through the global transaction processor, generates the global transaction number and the local transaction number corresponding to the real-time data updating operation, sends the global transaction number and the local transaction number to the data processing server, and receives the transaction number association information fed back by the data processing server. And under the condition that the transaction number association information is verified, pushing the local transaction number to a local transaction processor, and executing the distributed transaction according to the local transaction number by the local transaction processor to obtain distributed transaction processing data. Further, the data processing server polls the message queue and/or the cache queue according to the cache queue data mark, and sequentially obtains the distributed transaction data of each local transaction number under the global transaction number corresponding to the real-time data updating operation, so that the sequentially obtained distributed transaction data are synchronously updated to the aggregated database according to the transaction sequence, the problem that the existing real-time data updating method cannot guarantee the time sequence of real-time data updating is solved, the time sequence of real-time data updating can be guaranteed, and the accuracy and reliability of real-time data updating are improved.
It should be noted that any permutation and combination between the technical features in the above embodiments also belong to the scope of the present invention.
EXAMPLE six
Fig. 6 is a schematic diagram of a real-time data updating apparatus according to a sixth embodiment of the present invention, where the real-time data updating apparatus may be configured in a data processing server, and as shown in fig. 6, the apparatus includes: a distributed transaction data acquisition module 610 and a distributed transaction data update module 620, wherein:
a distributed transaction data obtaining module 610, configured to poll the target queue according to the cache queue data flag, so as to sequentially obtain distributed transaction data of each local transaction number under a global transaction number corresponding to a real-time data update operation; the target queue comprises a message queue and/or a buffer queue;
and a distributed transaction processing data updating module 620, configured to synchronously update the sequentially obtained distributed transaction processing data to the aggregation database according to the transaction processing sequence.
Optionally, the distributed transaction data obtaining module 610 is specifically configured to: polling the message queue to obtain distributed transaction processing data of the current target local transaction number according to the transaction processing sequence under the condition that the mark data of the cache queue data mark is determined to be first mark data;
polling the message queue and the cache queue to obtain distributed transaction processing data of the current target local transaction number according to the transaction processing sequence under the condition that the mark data of the cache queue data mark is determined to be second mark data;
and when the distributed transaction processing data of the current local transaction number acquired by polling the target queue is not the distributed transaction processing data of the current target local transaction number, the distributed transaction processing data of the current local transaction number is used for caching into the cache queue.
Optionally, the real-time data updating apparatus may further include: and the flag bit resetting module is used for resetting the flag data of the data flag of the cache queue into the first flag data.
According to the embodiment of the invention, the distributed transaction data of each local transaction number under the global transaction number corresponding to the real-time data updating operation is sequentially acquired by polling the message queue and/or the cache queue through the data processing server according to the cache queue data mark, and the sequentially acquired distributed transaction data is synchronously updated to the aggregation database according to the transaction processing sequence, so that the problem that the time sequence of real-time data updating cannot be ensured in the existing real-time data updating method is solved, the time sequence of real-time data updating can be ensured, and the accuracy and reliability of real-time data updating are improved.
The real-time data updating device can execute the real-time data updating method executed by the data processing server provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. For details of the technology that is not elaborately described in this embodiment, reference may be made to a real-time data updating method executed by a data processing server according to any embodiment of the present invention.
EXAMPLE seven
Fig. 7 is a schematic diagram of a real-time data updating apparatus according to a seventh embodiment of the present invention, where the real-time data updating apparatus may be configured in a local transaction processor, as shown in fig. 7, the apparatus includes: a local transaction number obtaining module 710, a distributed transaction data generating module 720 and a distributed transaction data pushing module 730, wherein:
a local transaction number obtaining module 710, configured to obtain a local transaction number issued by the global transaction processor according to the real-time data update operation;
a distributed transaction processing data generating module 720, configured to execute a distributed transaction according to the local transaction number to obtain distributed transaction processing data;
the distributed transaction processing data pushing module 730 is configured to push the distributed transaction processing data to a message queue for storage;
and the distributed transaction processing data is used for synchronously updating the data processing server to the aggregation server according to the transaction processing sequence.
According to the embodiment of the invention, the local transaction number issued by the global transaction processor according to the real-time data updating operation is acquired through the local transaction processor, the distributed transaction is executed according to the local transaction number, the distributed transaction data is obtained, and the distributed transaction data is pushed to the message queue for storage, so that the distributed transaction data is synchronously updated to the aggregation server by the data processing server according to the transaction sequence, the problem that the time sequence of real-time data updating cannot be ensured in the existing real-time data updating method is solved, the time sequence of real-time data updating can be ensured, and the accuracy and the reliability of real-time data updating are improved.
The real-time data updating device can execute the real-time data updating method executed by the local transaction processor provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. For the technical details not described in detail in this embodiment, reference may be made to the real-time data updating method performed by the local transaction processor according to any embodiment of the present invention.
Example eight
Fig. 8 is a schematic diagram of a real-time data updating apparatus according to an eighth embodiment of the present invention, where the real-time data updating apparatus may be configured in a global transaction processor, as shown in fig. 8, the apparatus includes: a transaction number generation module 810, a transaction number sending module 820 and a local transaction number pushing module 830, wherein:
the transaction number generating module 810 is configured to generate a global transaction number and a local transaction number corresponding to a real-time data update operation in response to the real-time data update operation;
a transaction number sending module 820, configured to send the global transaction number and the local transaction number corresponding to the real-time data update operation to a data processing server;
a local transaction number pushing module 830, configured to receive the transaction number related information fed back by the data processing server, and push the local transaction number to a local transaction processor when it is determined that the transaction number related information passes verification, so that the local transaction processor executes a distributed transaction according to the local transaction number to obtain distributed transaction processing data;
and the distributed transaction processing data is used for synchronously updating the data processing server to the aggregation server according to the transaction processing sequence.
The embodiment of the invention responds to the real-time data updating operation through the global transaction processor, generates the global transaction number and the local transaction number corresponding to the real-time data updating operation, sends the global transaction number and the local transaction number to the data processing server, and receives the transaction number association information fed back by the data processing server. And under the condition that the transaction number association information is verified, pushing the local transaction number to a local transaction processor, and executing the distributed transaction according to the local transaction number by the local transaction processor to obtain distributed transaction processing data. Further, the data processing server polls the message queue and/or the cache queue according to the cache queue data mark, and sequentially obtains the distributed transaction data of each local transaction number under the global transaction number corresponding to the real-time data updating operation, so that the sequentially obtained distributed transaction data are synchronously updated to the aggregated database according to the transaction sequence, the problem that the existing real-time data updating method cannot guarantee the time sequence of real-time data updating is solved, the time sequence of real-time data updating can be guaranteed, and the accuracy and reliability of real-time data updating are improved.
The real-time data updating device can execute the real-time data updating method executed by the global transaction processor provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. For the technical details that are not described in detail in this embodiment, reference may be made to the real-time data updating method performed by the global transaction processor according to any embodiment of the present invention.
Example nine
FIG. 9 illustrates a block diagram of an electronic device 10 that may be used to implement embodiments of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 9, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
Processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as the real-time data update method.
In some embodiments, the real-time data update method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the real-time data update method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the real-time data update method by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

Claims (10)

1. A real-time data updating method is applied to a data processing server and comprises the following steps:
polling the target queue according to the data mark of the cache queue to sequentially acquire distributed transaction processing data of each local transaction number under the global transaction number corresponding to the real-time data updating operation; the target queue comprises a message queue and/or a buffer queue;
and synchronously updating the distributed transaction processing data acquired in sequence to an aggregation database according to the transaction processing sequence.
2. The method of claim 1, wherein polling the target queue according to the buffer queue data flag to sequentially obtain the distributed transaction data of each local transaction number under the global transaction number corresponding to the real-time data update operation comprises:
polling the message queue to obtain distributed transaction processing data of the current target local transaction number according to the transaction processing sequence under the condition that the mark data of the data mark of the cache queue is determined to be first mark data;
polling the message queue and the cache queue to obtain distributed transaction processing data of the current target local transaction number according to the transaction processing sequence under the condition that the mark data of the cache queue data mark is determined to be second mark data;
and when the distributed transaction processing data of the current local transaction number acquired by polling the target queue is not the distributed transaction processing data of the current target local transaction number, the distributed transaction processing data of the current local transaction number is used for caching into the cache queue.
3. The method of claim 2, further comprising, after synchronously updating the sequentially obtained distributed transaction data to an aggregation database in transaction order:
resetting the flag data of the buffer queue data flag to the first flag data.
4. A real-time data updating method applied to a local transaction processor comprises the following steps:
acquiring a local transaction number issued by the global transaction processor according to real-time data updating operation;
executing the distributed transaction according to the local transaction number to obtain distributed transaction processing data;
pushing the distributed transaction processing data to a message queue for storage;
and the distributed transaction processing data is used for synchronously updating the data processing server to the aggregation server according to the transaction processing sequence.
5. A real-time data updating method applied to a global transaction processor comprises the following steps:
responding to a real-time data updating operation, and generating a global transaction number and a local transaction number corresponding to the real-time data updating operation;
sending the global transaction number and the local transaction number corresponding to the real-time data updating operation to a data processing server;
receiving transaction number association information fed back by the data processing server, and pushing the local transaction number to a local transaction processor under the condition that the transaction number association information is confirmed to pass verification, so that the local transaction processor executes a distributed transaction according to the local transaction number, and distributed transaction processing data are obtained;
and the distributed transaction processing data is used for synchronously updating the data processing server to the aggregation server according to the transaction processing sequence.
6. A real-time data update apparatus, configured in a data processing server, comprising:
the distributed transaction processing data acquisition module is used for polling the target queue according to the data mark of the cache queue so as to sequentially acquire distributed transaction processing data of each local transaction number under the global transaction number corresponding to the real-time data updating operation; the target queue comprises a message queue and/or a buffer queue;
and the distributed transaction processing data updating module is used for synchronously updating the distributed transaction processing data acquired in sequence to the aggregation database according to the transaction processing sequence.
7. A real-time data update apparatus configured for a local transaction processor, comprising:
the local transaction number acquisition module is used for acquiring a local transaction number issued by the global transaction processor according to the real-time data updating operation;
the distributed transaction processing data generation module is used for executing the distributed transaction according to the local transaction number to obtain distributed transaction processing data;
the distributed transaction processing data pushing module is used for pushing the distributed transaction processing data to a message queue for storage;
and the distributed transaction processing data is used for synchronously updating the data processing server to the aggregation server according to the transaction processing sequence.
8. A real-time data update apparatus configured in a global transaction processor, comprising:
the transaction number generation module is used for responding to real-time data updating operation and generating a global transaction number and a local transaction number corresponding to the real-time data updating operation;
the transaction number sending module is used for sending the global transaction number and the local transaction number corresponding to the real-time data updating operation to a data processing server;
the local transaction number pushing module is used for receiving the transaction number associated information fed back by the data processing server, and pushing the local transaction number to a local transaction processor under the condition that the transaction number associated information is verified, so that the local transaction processor executes a distributed transaction according to the local transaction number to obtain distributed transaction processing data;
and the distributed transaction processing data is used for synchronously updating the data processing server to the aggregation server according to the transaction processing sequence.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the first and the second end of the pipe are connected with each other,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the real-time data update method of any one of claims 1-3, 4 or 5.
10. A computer storage medium storing computer instructions for causing a processor to perform the real-time data update method of any one of claims 1-3, 4 or 5 when executed.
CN202211209286.5A 2022-09-30 2022-09-30 Real-time data updating method and device, electronic equipment and storage medium Pending CN115525666A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211209286.5A CN115525666A (en) 2022-09-30 2022-09-30 Real-time data updating method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211209286.5A CN115525666A (en) 2022-09-30 2022-09-30 Real-time data updating method and device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN115525666A true CN115525666A (en) 2022-12-27

Family

ID=84699462

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211209286.5A Pending CN115525666A (en) 2022-09-30 2022-09-30 Real-time data updating method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN115525666A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115827261A (en) * 2023-01-10 2023-03-21 北京燧原智能科技有限公司 Data synchronization method, device, server and medium based on distributed network

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115827261A (en) * 2023-01-10 2023-03-21 北京燧原智能科技有限公司 Data synchronization method, device, server and medium based on distributed network

Similar Documents

Publication Publication Date Title
CN113193947B (en) Method, apparatus, medium, and program product for implementing distributed global ordering
CN109783151B (en) Method and device for rule change
CN115525666A (en) Real-time data updating method and device, electronic equipment and storage medium
CN116383207A (en) Data tag management method and device, electronic equipment and storage medium
CN114564318A (en) Data processing method, device and system, electronic equipment and storage medium
CN115529186A (en) SSL certificate unloading method, device and system based on soft load balancing
CN115454971A (en) Data migration method and device, electronic equipment and storage medium
CN115391158A (en) Time delay determination method, system and device and electronic equipment
CN114579260A (en) Transaction processing method and system
CN114579311A (en) Method, apparatus, device and storage medium for executing distributed computing task
CN112883041A (en) Data updating method and device, electronic equipment and storage medium
CN113961641A (en) Database synchronization method, device, equipment and storage medium
CN114553894B (en) Data synchronization method, device, system and storage medium
CN116974825A (en) Backup method, device, equipment and storage medium
CN114924806B (en) Dynamic synchronization method, device, equipment and medium for configuration information
CN116016265B (en) Message all-link monitoring method, device, system, equipment and storage medium
CN118055068A (en) Message processing method, device, equipment and medium based on DPDK
CN115858104A (en) Method, device and equipment for acquiring snapshot of transaction commit sequence number and storage medium
CN115203139A (en) Log query method and device, electronic equipment and storage medium
CN117667144A (en) Annotating hot refreshing method, annotating hot refreshing device, annotating hot refreshing equipment and annotating hot refreshing medium
CN116302371A (en) Transaction visibility judging method, device, equipment and storage medium
CN114416881A (en) Real-time synchronization method, device, equipment and medium for multi-source data
CN116431928A (en) Time sequence data pushing method, device, equipment and storage medium
CN115730000A (en) Medical data integration method, device, equipment and medium based on data lake
CN116015665A (en) Method, device, equipment and medium for repeatedly detecting virtual article

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination