CN117785908A

CN117785908A - Method and device for executing data processing request

Info

Publication number: CN117785908A
Application number: CN202311617227.6A
Authority: CN
Inventors: 吴长斌
Original assignee: Industrial and Commercial Bank of China Ltd ICBC; ICBC Technology Co Ltd
Current assignee: Industrial and Commercial Bank of China Ltd ICBC; ICBC Technology Co Ltd
Priority date: 2023-11-29
Filing date: 2023-11-29
Publication date: 2024-03-29

Abstract

The invention discloses a method and a device for executing a data processing request, wherein the method comprises the following steps: generating a data query statement based on the received data processing request of the user; determining a target database for executing the data query statement; determining the current offset of a target database from preset database offset information; when the current offset is different from the offsets of other main databases, acquiring offset information and log files of the other main databases, updating data of a target database, and updating the current offset of the target database; repeating the steps until the current offset is the same as the offsets of other main databases; executing the data query statement with the updated target database to obtain data query information; and sending out the data query information. The method and the device are used for improving the efficiency and the accuracy of database updating and improving the timeliness and the efficiency of executing the data processing request.

Description

Method and device for executing data processing request

Technical Field

The present invention relates to the field of artificial intelligence technologies, and in particular, to a method and an apparatus for executing a data processing request.

Background

This section is intended to provide a background or context to the embodiments of the invention that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

Because of the increasing demands of high availability of platforms, the private network platform uses a main machine room and a standby machine room for application support, and part of main databases provide data synchronization plug-ins, such as Mysql databases, and data synchronization is performed by using binlog. However, at present, all main databases basically have the condition of inconsistent data synchronization.

Because the problem of data consistency under the condition of network abnormality is not solved, only a single database is served externally at the same time, so that the database can be recovered only by manual intervention under the condition of abnormality, and the availability of a platform is greatly reduced. Other databases, such as neo4j community version, do not provide database synchronization functionality other than MySql.

Disclosure of Invention

The embodiment of the invention provides a data processing request execution method, which is used for improving the efficiency and accuracy of database updating and improving the timeliness and efficiency of executing data processing requests, and comprises the following steps:

generating a data query statement based on the received data processing request of the user; determining a target database for executing the data query statement;

Determining the current offset of a target database from preset database offset information; when the current offset is different from the offsets of other main databases, acquiring offset information and log files of the other main databases; the database offset information comprises offset information of different main databases; the offset information is used for representing update information of the main database; the log file is used for updating data of different data tables in the main database;

according to the offset information and the log file of the other main databases, carrying out data updating on a target database, and updating the current offset of the target database; repeating the steps until the current offset is the same as the offsets of other main databases;

executing the data query statement with the updated target database to obtain data query information; and sending out the data query information.

The embodiment of the invention also provides a device for executing the data processing request, which is used for improving the efficiency and the accuracy of updating the database and improving the timeliness and the efficiency of executing the data processing request, and the device comprises:

the data query statement generation module is used for generating a data query statement based on a received data processing request of a user; determining a target database for executing the data query statement;

The offset determining module is used for determining the current offset of the target database from preset database offset information; when the current offset is different from the offsets of other main databases, acquiring offset information and log files of the other main databases; the database offset information comprises offset information of different main databases; the offset information is used for representing update information of the main database; the log file is used for updating data of different data tables in the main database;

the data updating module is used for carrying out data updating on the target database according to the offset information and the log file of the other main databases and updating the current offset of the target database; repeating the steps until the current offset is the same as the offsets of other main databases;

the data query statement execution module is used for executing the data query statement by the updated target database to obtain data query information; and sending out the data query information.

The embodiment of the invention also provides computer equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the data processing request execution method when executing the computer program.

The embodiment of the invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program realizes the data processing request execution method when being executed by a processor.

The embodiment of the invention also provides a computer program product, which comprises a computer program, and the computer program realizes the data processing request execution method when being executed by a processor.

In the embodiment of the invention, a data query statement is generated based on a received data processing request of a user; determining a target database for executing the data query statement; determining the current offset of a target database from preset database offset information; when the current offset is different from the offsets of other main databases, acquiring offset information and log files of the other main databases; the database offset information comprises offset information of different main databases; the offset information is used for representing update information of the main database; the log file is used for updating data of different data tables in the main database; according to the offset information and the log file of the other main databases, carrying out data updating on a target database, and updating the current offset of the target database; repeating the steps until the current offset is the same as the offsets of other main databases; executing the data query statement with the updated target database to obtain data query information; compared with the technical scheme that in the prior art, when a single database is abnormal, data is not the latest data and manual operation is needed, the method and the device automatically realize data updating of the target database by acquiring offset information of the target database and other main databases and combining log files of the other main databases, and do not need an updating process of the target database manually, so that timeliness of the database updating is improved, data processing requests can be normally processed without manual intervention immediately when the database is abnormal, and fault tolerance of data request processing is greatly improved; meanwhile, the problems of database error update and delayed update which are caused by manual operation are avoided, and the efficiency and the accuracy of database update are improved; furthermore, timeliness and efficiency of data processing requests are improved.

Drawings

In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. In the drawings:

FIG. 1 is a flow chart of a method for executing a data processing request according to an embodiment of the invention;

FIG. 2 is a diagram illustrating a method for executing a data processing request according to an embodiment of the present invention;

FIG. 3 is a diagram illustrating an exemplary configuration of a data processing request execution device according to an embodiment of the present invention;

FIG. 4 is a diagram illustrating an embodiment of a data processing request execution device according to the present invention;

fig. 5 is a schematic diagram of a computer device according to an embodiment of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention will be described in further detail with reference to the accompanying drawings. The exemplary embodiments of the present invention and their descriptions herein are for the purpose of explaining the present invention, but are not to be construed as limiting the invention.

The term "and/or" is used herein to describe only one relationship, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

In the description of the present specification, the terms "comprising," "including," "having," "containing," and the like are open-ended terms, meaning including, but not limited to. Reference to the terms "one embodiment," "a particular embodiment," "some embodiments," "for example," etc., means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. The sequence of steps involved in the embodiments is used to schematically illustrate the practice of the present application, and is not limited thereto and may be appropriately adjusted as desired.

The data acquisition, storage, use, processing and the like in the technical scheme meet the relevant regulations of national laws and regulations.

In order to meet the increasing high availability demands of the platform, the private network platform adopts a main machine room and a standby machine room for application support. To ensure consistency and reliability of the data, part of the master database provides data synchronization plug-ins, such as MySQL database, that utilize binlog for data synchronization. However, all the main databases have inconsistent data synchronization, which is a problem to be solved urgently.

In order to solve this problem, intensive research and discussion is required. First, for the problem of inconsistent data synchronization, we can consider using more advanced algorithms and tools to improve the accuracy and reliability of data synchronization. For example, using a distributed database system or introducing more advanced data synchronization techniques, such as Paxos-based distributed consistency algorithms, etc.

Second, for data consistency problems caused by network anomalies, we can attempt to introduce fault tolerance mechanisms and backup mechanisms to ensure the integrity of the data. For example, using a data backup scheme based on network communications, the backup database may automatically take over and provide services when an anomaly occurs in the primary database. Furthermore, we can also employ a distributed database architecture to avoid the impact of single point failures on data consistency.

Other databases, such as Neo4j community version, do not provide database synchronization functionality other than MySQL. The current synchronous plug-in lacks an active early warning mechanism, so that the synchronous plug-in is difficult to find in time once the synchronous abnormality occurs, and manual investigation is needed.

To solve these problems, the following points can be considered: first, for the problem of lack of an active early warning mechanism, a monitoring system and an alarm mechanism can be introduced to discover and solve the potential problem in time. For example, a monitoring index and a threshold are set, and when the data synchronization is abnormal, the system automatically sends out an alarm notification and reminds an administrator of timely processing. Second, for the problem of lack of synchronization functionality in other databases, such as Neo4j community versions, it may be considered to use third party tools or self-development to implement data synchronization functionality. Finally, to increase the availability of the overall system, the introduction of load balancing and fault tolerant mechanisms may also be considered to ensure that the system is able to operate stably and provide services.

In order to solve the above problems, an embodiment of the present invention provides a method for executing a data processing request, which is used to improve efficiency and accuracy of database update, and improve timeliness and efficiency of executing the data processing request, referring to fig. 1, the method may include:

Step 101: generating a data query statement based on the received data processing request of the user; determining a target database for executing the data query statement;

step 102: determining the current offset of a target database from preset database offset information; when the current offset is different from the offsets of other main databases, acquiring offset information and log files of the other main databases; the database offset information comprises offset information of different main databases; the offset information is used for representing update information of the main database; the log file is used for updating data of different data tables in the main database;

step 103: according to the offset information and the log file of the other main databases, carrying out data updating on a target database, and updating the current offset of the target database; repeating the steps until the current offset is the same as the offsets of other main databases;

step 104: executing the data query statement with the updated target database to obtain data query information; and sending out the data query information.

In the method, the database offset information and the log file are used for updating the data of the target database, so that the efficiency and the accuracy of updating the database can be effectively improved, and the timeliness and the efficiency of executing the data processing request can be improved. Specifically, by acquiring offset information and log files of other main databases and updating data of the target database, the accuracy and consistency of data updating can be ensured. Meanwhile, the integrity of data update can be ensured by repeatedly executing the steps until the current offset is the same as the offsets of other main databases. And finally, executing the data query statement by using the updated target database, obtaining and sending out the data query information, and facilitating the subsequent data processing and use of a user.

The method has the characteristics of simplicity, easiness in implementation and suitability for various scenes needing database updating. Meanwhile, due to the fact that the database offset information and the log file are used, the problems of efficiency and accuracy existing in a traditional database updating method can be effectively solved, and timeliness and efficiency of data updating are improved. In addition, the method can realize synchronous update of the data of a plurality of main databases, and further ensure the integrity and consistency of the data update.

In the embodiment of the invention, a data query statement is generated based on a received data processing request of a user; determining a target database for executing the data query statement; determining the current offset of a target database from preset database offset information; when the current offset is different from the offsets of other main databases, acquiring offset information and log files of the other main databases; the database offset information comprises offset information of different main databases; the offset information is used for representing update information of the main database; the log file is used for updating data of different data tables in the main database; according to the offset information and the log file of the other main databases, carrying out data updating on a target database, and updating the current offset of the target database; repeating the steps until the current offset is the same as the offsets of other main databases; executing the data query statement with the updated target database to obtain data query information; and sending out the data query information. Compared with the technical scheme that in the prior art, the single database is abnormal, so that data is not updated and manual operation is needed, the data updating of the target database is automatically realized by acquiring offset information of the target database and other main databases and combining with log files of other main databases, the updating process of the target database is not needed, the timeliness of the database updating is improved, the data processing request can still be normally processed without manual intervention immediately when the database is abnormal, and the fault tolerance rate of data request processing is greatly improved. Meanwhile, the problems of database error update and delayed update which are caused by unavoidable manual operation are avoided, and the efficiency and the accuracy of database update are improved. Furthermore, timeliness and efficiency of data processing requests are improved.

The embodiment of the invention also provides a specific implementation mode of the data updating method. The method comprises the following steps:

1. and receiving a data processing request of a user, and generating a corresponding data query statement according to the request.

2. A target database is determined for which the data query statement is to be executed.

3. And determining the current offset of the target database from preset database offset information.

4. And if the current offset is different from the offsets of other main databases, acquiring the offset information and the log file of the other main databases.

5. And according to the offset information and the log file of the other main databases, carrying out data updating on the target database, and updating the current offset of the target database.

6. Steps 4 and 5 are repeated until the current offset is the same as the offset of the other master database.

7. And executing the data query statement with the updated target database to obtain the data query information.

8. And sending out data query information.

By the method, the data of the target database can be automatically updated, the updating process of the target database by manpower is not needed, the timeliness of the database updating is improved, the data processing request can be normally processed without manual intervention immediately when the database is abnormal, and the fault tolerance of the data request processing is greatly improved. Meanwhile, the problems of database error update and delayed update which are caused by unavoidable manual operation are avoided, and the efficiency and the accuracy of database update are improved. Furthermore, timeliness and efficiency of data processing requests are improved.

Based on the above embodiments, another implementation of the present invention is provided. Specifically, based on a received data processing request of a user, a data query statement may be generated and a target database executing the data query statement is determined. And then, automatically updating the data of the target database by acquiring offset information of the target database and other main databases and combining log files of the other main databases. And repeatedly executing the steps until the current offset is the same as the offsets of other main databases. And finally, executing the data query statement with the updated target database to obtain data query information, and sending the data query information.

Compared with the prior art, the implementation mode has the following advantages: firstly, the data of the target database is automatically updated by acquiring offset information of the target database and other main databases and combining log files of the other main databases, so that the updating process of the target database is not needed manually, and the timeliness of database updating is improved. And secondly, when the database is abnormal, the data processing request can still be normally processed without manual intervention, and the fault tolerance rate of data request processing is greatly improved. Meanwhile, the problems of database error update and delayed update, which are caused by unavoidable manual operation, are avoided, and the efficiency and the accuracy of database update are improved. Finally, timeliness and efficiency of the data processing request are improved.

In the embodiment of the invention, the data query statement may include an SQL statement, a NoSQL statement or other query statement of structured or unstructured data. The target database may be a relational database, a non-relational database, a distributed database, or a cloud database. The offset information may include a timestamp, version number, or other identification that characterizes the primary database update information. The log files may include operation logs, change logs, or other log files capable of recording data update processes.

Further, the step of obtaining offset information and log files of the other master databases may include: obtaining offset information from other main databases; obtaining log files from other main databases; or obtaining the offset information and the log file of other main databases from preset storage positions. The step of updating the data of the target database may include: comparing and updating the data in the target database according to the offset information and the log file of the other main databases; or copying the data in the other main databases to the target database.

In an embodiment of the present invention, the data query statement may include an SQL statement, a NoSQL statement, or other query statement of structured or unstructured data. The target database may be a relational database, a non-relational database, a distributed database, or a cloud database. The offset information may include a timestamp, version number, or other identification that characterizes the primary database update information. The log files may include operation logs, change logs, or other log files capable of recording data update processes.

In the embodiment of the invention, the data query statement may include an SQL statement, a NoSQL statement or other query statement of structured or unstructured data, which enables the embodiment of the invention to adapt to various different data sources and data types. The target database may be a relational database, a non-relational database, a distributed database, or a cloud database, which enables embodiments of the present invention to be applied to a variety of different database systems and data storage modes. The offset information may include a timestamp, version number, or other identification that characterizes the primary database update information, which enables embodiments of the present invention to accurately track and synchronize updates of the respective primary databases. The log file may include an operation log, a change log, or other log file capable of recording data update processes, which enables embodiments of the present invention to record data update histories and processes in detail.

Further, the step of obtaining offset information and log files of the other master databases may include: obtaining offset information from other main databases; obtaining log files from other main databases; or obtaining the offset information and the log file of other main databases from preset storage positions. The embodiment of the invention can flexibly acquire the needed data synchronization information and effectively reduce the time and resource consumption in the data synchronization process. The step of updating the data of the target database may include: comparing and updating the data in the target database according to the offset information and the log file of the other main databases; or copying the data in the other main databases to the target database. The embodiment of the invention can accurately synchronize the data updating of each main database and can maintain the data consistency and the integrity of the target database.

Furthermore, the step of issuing the data query information may include: transmitting the data query information to a user; storing the data query information into a target database; or the data inquiry information is sent to another data processing system for processing. The data query information may include query results, query status, or other information related to the data processing request.

When the method is implemented, firstly, a data query statement is generated based on a received data processing request of a user; and determining a target database for executing the data query statement.

In an embodiment, generating a data query statement based on a received data processing request of a user includes:

generating a data query statement based on a data processing request of a user received from an application cluster; the application cluster comprises an application for executing the data processing request; the application is used for inquiring the data corresponding to the data inquiry statement in the corresponding cache cluster;

determining a target database for executing the data query statement, comprising:

and when the data does not exist in the cache cluster, determining a target database for executing the data query statement.

In a specific implementation, a data query statement is first generated based on a received data processing request of a user. This process may be implemented by some data processing tool or script, such as programming a script in a programming language such as Python or Java, receiving a user's data processing request, and then converting the request into a data query statement.

After the data query statement is generated, a target database for executing the data query statement needs to be determined. In an embodiment, this target database may be any database that may execute data query statements, such as MySQL, postgreSQL, oracle, etc.

After the target database is determined, the data query statement may be executed. This process may be implemented through interfaces such as database connectors or JDBC. When executing a data query statement, the query result needs to be returned to the user. This process of returning the results may be implemented via HTTP or other network protocols.

If there is no data in the cache cluster corresponding to the data query statement, then a target database for executing the data query statement needs to be determined. This process may be implemented by some cache-elimination policies, such as LRU (Least Recently Used) policies, etc. After determining the target database, the data query statement may be executed via an interface such as a connector or JDBC. When executing the data query statement, the query result needs to be stored in the cache cluster, so that the result can be directly obtained from the cache when querying next time, and the query efficiency is improved.

In the whole process, operations such as encryption, decryption, verification and the like are required to be carried out on the data so as to ensure the safety and the integrity of the data. Meanwhile, the whole process needs to be monitored and managed so as to discover problems in time and process the problems.

In an embodiment, when the user performs service processing, the user may directly send data processing request information to the application cluster app.

And then, a preset load balancing server distributes the request to the middleware of an application in the proxy application cluster according to the load conditions of different applications in the application cluster. And after receiving the request, the application processes the request and sends the operation request database language to the data operation middleware.

In one embodiment, the target database may be a newly generated database. The newly generated database can be obtained by a developer through executing a database generation script, and the script can be obtained through a data operation middleware and simultaneously increase the offset information key of a data table in the script in the data offset table. This key is used to identify the current synchronization offset for different databases in the database cluster for ease of administration and operation.

In another embodiment, the target database may also be an existing database for storing and managing daily operational data of the enterprise. The database can be synchronized with the source database in real time through the data synchronizing tool, so that the real-time property and accuracy of the data are ensured.

After the target database is generated, a developer can execute the data migration script through the data operation middleware to migrate the data in the source database into the target database. In the migration process, the data operation middleware automatically tracks the progress of data migration and feeds progress information back to a developer.

Meanwhile, in order to ensure the accuracy and the integrity of data migration, developers can also verify the data in the source database and the target database through a data verification tool. If any inconsistent data is found, the developer can repair via the data repair tool. By using the target database and the data operation middleware, the database can be conveniently managed and operated, and the safety and reliability of the data are improved. Meanwhile, by the application of the data synchronization tool and the data verification tool, the real-time performance and accuracy of data can be guaranteed, and better data support and management are provided for enterprises.

As a specific example, the database is generated as follows: first, a developer needs to write and execute a database generation script. The script includes operations such as creating a data table, and the data operation middleware adds the offset information key of the data table in the script to the data offset table simultaneously when the operations are executed. This key is the current synchronization offset used to identify the different databases in the database cluster.

And the initial offset storage information of the target database may be represented as follows: { key: "t1" [ a1:0, b1:0] }. Where key is the data table in the newly added target database and a1 and b1 are the databases in the cluster, the values identifying the current latest synchronization position of the library.

In addition, an entry may be added to the table offset self-increment sequence (e.g., 1,2,3, 4.) of the target data table in the data offset cache cluster. This self-increment sequence may be invoked by the INCR command of Redis to obtain the latest offset data and identify it as the current latest data offset. By doing so, the real-time performance and the accuracy of the data can be ensured, so that the database cluster is better managed and operated.

In the process of initializing the target database, since only initialization data is in the system of the target database, the offset of the initial table in the offset clusters of all data tables is 0.

After the steps are completed, a developer can execute a database generation script and add corresponding offset information in a data offset table. Specifically, a developer may write a script to read the offset information in the data offset table and generate a corresponding database table based on the offset information. In this process, the data manipulation middleware automatically records the offset information of each database table for subsequent data synchronization operations.

Once the initialization process of the target database is completed, only initialization data is in the system of the target database, wherein the offset of the initial table in the offset cluster of all data tables is 0. At this point, the developer may recall the self-increasing sequence in the data offset cache cluster to generate a new offset entry for each data table. This new offset entry may be used to identify the latest synchronization position of the table in the database cluster.

After these steps are completed, the developer can begin subsequent data synchronization operations. Specifically, when new data needs to be synchronized to the target database, a developer can call the relevant interface of the data operation middleware to write the data into the target database. In the process, the data operation middleware automatically determines which data need to be synchronized according to the current offset information of the data table, so that the real-time performance and accuracy of the data are ensured.

Meanwhile, in order to better manage and operate the database clusters, the developer can also update the offset information of each database in the data offset table in real time. Thus, a developer can know the latest synchronous position of each database at any time so as to perform data synchronization and verification operation in time. Meanwhile, by using the Redis and other caching technologies, the efficiency and performance of data operation can be improved. Through the above steps, the developer can successfully generate a target database containing initialization data and the latest synchronization position identification. Meanwhile, by using technical means such as a data offset table, a data offset cache cluster and the like, the database cluster can be managed and operated better, and the real-time property and accuracy of data are ensured.

In the specific implementation, generating a data query statement based on a received data processing request of a user; after determining a target database for executing the data query statement, determining the current offset of the target database from preset database offset information; when the current offset is different from the offsets of other main databases, acquiring offset information and log files of the other main databases; the database offset information comprises offset information of different main databases; the offset information is used for representing update information of the main database; the log file is used for updating data of different data tables in the main database.

In one embodiment, the preset database offset information stores offset information of different data tables of different main databases.

In the implementation stage, the system generates corresponding data query sentences according to the received user data processing request. Then, a target database is determined to execute the data query statement. Then, the system determines the current offset of the target database from the preset database offset information. If the current offset is different from the offsets of the other master databases, the system will acquire the offset information and log files of the other master databases. These database offset information include offset information for different tables of different master databases, used to characterize the update information of the master database. The log file is used for recording data updates of different data tables in the main database. These log files are critical to maintaining consistency and integrity of the database.

In the implementation process, the system can query and process the data of the target database according to the preset database offset information. If the offset of the target database is found to be different from that of other main databases, the system acquires the offset information and log files of the other main databases so as to know the updating condition of different data tables. This information will be used to update the target database to keep it synchronized with the other master databases.

In addition, some details need to be paid attention to in the implementation process. For example, for large databases, techniques such as paging queries may be required to improve query efficiency. In addition, in order to prevent data inconsistency, backup and restore operations for the database are required. In addition, in order to ensure the security of data, it is necessary to perform encryption processing on the data. By the above embodiment, the database synchronization method based on the offset can be realized. The method can effectively solve the problem of data synchronization between different main databases and improve the data processing efficiency and accuracy. Meanwhile, by using the log file and the offset information, the consistency and the integrity of the data can be ensured.

In one embodiment, the log file is used to: storing data information of a current database, wherein the data information comprises offset data and SQL sentences; the offset information is used to: the database operation middleware judges whether the data in the current database is up to date, whether the current operation command can be executed, and executing standard current offset = in-table stored offset +1.

In one embodiment, the log file is a key auxiliary tool for storing data information of the current database, including detailed data entries and various data offset information. These offset data are critical to database operations because they provide accurate information about the location of the data in memory and the status of the data update. In addition, the log file also contains SQL statements that describe various operations performed on the database.

Regarding the offset information, these information are mainly used to determine whether the database operating middleware can determine whether the data in the current database is up-to-date. The offset information may also help the middleware determine the current data state, determine whether the current operation command can be executed. In the execution standard, the current offset is set equal to the in-table stored offset plus 1 to ensure continuity and accuracy of data. This standardized operational flow ensures the correctness and consistency of database operations.

When the method is implemented, after the offset information and the log file of the other main databases are obtained, the target database is subjected to data updating according to the offset information and the log file of the other main databases, and the current offset of the target database is updated; and repeatedly executing the steps until the current offset is the same as the offsets of other main databases.

In an embodiment, according to the offset information and the log file of the other main databases, data updating is performed on a target database, and the current offset of the target database is updated; and then, when the updated current offset is different from the offsets of other main databases, repeating the steps of updating the data of the target database and updating the current offset of the target database until the current offset is the same as the offsets of other main databases.

In one embodiment, the data query statement corresponds to a target data table in a target database;

determining the current offset of the target database from preset database offset information comprises the following steps:

determining the current offset of a target data table of a target database from preset database offset information;

When the current offset is different from the offsets of other main databases, acquiring the offset information and log files of the other main databases, including:

when the current offset of the target data table is different from the offset of the target data table in other main databases, acquiring the offset information and log files of the target data table in the other main databases;

according to the offset information and the log file of the other main databases, carrying out data updating on the target database and updating the current offset of the target database, wherein the method comprises the following steps:

and according to the offset information and the log file of the target data table in the other main databases, carrying out data updating on the target data table in the target database, and updating the current offset of the target data table of the target database.

In the above embodiment, the data query statement may be used to query data in a target data table in the target database, and the data in the target data table may be used to represent information of the target object. When determining the current offset of the target database, the preset database offset information may be used to determine the current offset of the target data table of the target database. When the current offset of the target data table is different from the offset of the target data table in other main databases, the offset information and the log file of the target data table in the other main databases can be obtained, and the target data table in the target database is updated according to the offset information and the log file of the other main databases, and the current offset of the target data table of the target database is updated.

In the above embodiment, the other main databases may be the same type of databases or different types of databases. The log file may record offset information and data update records of the target data table in the other master database. The data update may be a delta update or a full update.

In a modified embodiment, the data query statement may be used to query data in a plurality of target data tables in a plurality of target databases, and the data in each of the target data tables may be used to represent information for a different type of object. When determining the current offsets of the plurality of target databases, the current offset of the target data table of each target database may be determined using the preset plurality of database offset information. When the current offset of each target data table is different from the offsets of the target data tables in other main databases, the offset information and the log file of the target data table in the other main databases can be obtained, and according to the offset information and the log file of the other main databases, the data updating is performed on a plurality of target data tables in a plurality of target databases, and the current offsets of a plurality of target data tables in the plurality of target databases are updated.

In the modified embodiment, the other main databases may be the same type of databases or different types of databases. The log file may record offset information and data update records for each target data table in the other master database. The data update may be a delta update or a full update.

In the above embodiment, the current offset of the target data table of the target database is determined from the preset database offset information, and the target data table in the target database is updated according to the offset information and the log file of the target data table in the other main databases, so that the dimension of the data table in the database can complete the processing of the data processing request, which is a further refinement operation of the database updating processing in the present invention.

In one embodiment, further comprising:

when the data updating of the target database fails for a plurality of times, sending out alarm information representing the data updating of the target database for a plurality of times.

In this embodiment, when the data update of the target database fails multiple times, the system triggers an alarm mechanism to send alarm information indicating that the data update of the target database fails multiple times. The alert information may be in the form of an email, a short message, a telephone voice, or a system message, etc., to allow an administrator or related personnel to timely learn and handle the problem.

The design of such an alert mechanism aims to improve the reliability and stability of the system. When the data updating of the target database fails for a plurality of times, the system can send out alarm information in time, so that an administrator or related personnel can intervene in time and solve the problem, and the problem that the data updating failure has larger influence on the whole system is avoided.

In addition, the embodiment also provides an intelligent alarm mechanism. The system may sort and prioritize the alert information based on the type and severity of the alert information so that an administrator or related personnel can prioritize the more urgent and important alert information. The intelligent alarm mechanism can improve the response speed and the processing efficiency of the system, thereby better ensuring the normal operation of the system.

In the embodiment, when the data updating of the target database fails for a plurality of times, the alarm information representing the data updating of the target database fails for a plurality of times is sent out, so that an early warning mechanism is provided, synchronization abnormality can be timely pushed to related personnel, and problems can be timely found.

When the method is implemented, the steps are repeatedly executed until the current offset is the same as the offsets of other main databases, and the data query statement is executed by the updated target database to obtain data query information; and sending out the data query information.

In the embodiment, the data query statement is executed by the updated target database to obtain the data query information, so that the data processing request can be processed finally.

In the implementation process, the above steps need to be repeatedly performed until the current offset is consistent with the offsets of other main databases. And then, executing the data query statement by using the updated target database to acquire the data query information. And finally, issuing the acquired data query information.

In an embodiment, the data processing request can be satisfactorily processed by performing a data query statement using the updated target database to obtain the data query information. The scheme has good compatibility, can be compatible with various databases, and is not limited by the synchronous function provided by the databases. This provides the possibility to solve the synchronization problem that some different types of databases do not provide a synchronization scheme.

In addition, the scheme has expandability. With the development of service scale and the increase of data volume, the scheme can be easily expanded to more database types and larger data volumes. Meanwhile, the scheme also has good stability and reliability, and can ensure the accuracy and the integrity of data processing.

A specific example is given below, as shown in fig. 2, to illustrate a specific application of the method of the present invention.

In this embodiment, taking a data operation middleware as an example, the present offset of the target database is determined from preset database offset information in the present invention; when the current offset is different from the offsets of other main databases, acquiring offset information and log files of the other main databases; the database offset information comprises offset information of different main databases; the offset information is used for representing update information of the main database; the log file is used for updating data of different data tables in the main database; according to the offset information and the log file of the other main databases, carrying out data updating on a target database, and updating the current offset of the target database; repeating the steps until the current offset is the same as the offsets of other main databases. The invention is not limited thereto, and any tool that can implement the method of the invention falls within the scope of the invention.

S1: when executing the database generation script, the developer should add the data offset information key to the data offset table through the data operation middleware at the same time so as to identify the current synchronous offset of different databases in the database cluster. The initial offset storage information should include a key representing the newly added database table, and two values a1 and b1 representing the current latest synchronization position of the databases in the cluster. Meanwhile, the table offset self-increasing queue should be newly added in the data offset cache cluster to be used as the identification of the current latest data offset

In the process of executing a database generation script by a developer, an important step is to add a data offset information key to a data offset table through a data operation middleware. This step is to identify the current synchronization offsets for the different databases in the database cluster. The initial offset storage information should include a key representing the newly added database table, and two values a1 and b1 representing the current latest synchronization position of the databases in the cluster. Meanwhile, the table offset self-increasing queue should be newly added in the data offset cache cluster to be used as the identification of the current latest data offset.

In the data manipulation middleware, a developer needs to write corresponding codes to realize the storage and the reading of the data offset. When a developer executes a database generation script, the script stores the data offset information into a data offset table. In this process, the developer needs to use the middleware-provided API to operate the data offset table. In particular, the developer needs to call a storage function provided by the middleware, and pass data offset information to the function as parameters. The function stores the data offset information in the data offset table and returns a message that the storage was successful.

In the data offset table, each data offset information includes a unique key and corresponding values a1 and b 1. key is a unique identifier for identifying each data offset, and a1 and b1 represent the current synchronization offsets for the different databases in the database cluster. The developer needs to set the values of a1 and b1 according to the actual situation so as to ensure that the data in the database cluster are synchronous accurately.

In addition, in order to facilitate tracking and managing data offsets, a developer needs to newly add the table offset self-increasing queue in the data offset cache cluster. This queue will record the identity of the current latest data offset so that subsequent data operations can be referenced and used. Through the queue, a developer can quickly acquire the latest data offset information, so that the data synchronization process of the database cluster is better controlled.

S2: in the system initial state, since only the initialization data is used, the offsets of all the data tables should be initialized to 0 in the cluster. This means that at the beginning of the system operation, the offsets of all database tables are set to 0 to ensure an initial state of synchronization.

As time goes by, the records in the data table will increase and the offset will increase. However, in some cases, it may be necessary to reset the offset back to 0 in order to resynchronize the data.

For example, at system upgrades or data migration, the data tables may need to be reinitialized. At this point, the offsets of all data tables may be reset to 0 in the cluster to ensure that all nodes are in sync.

In addition, if the system fails or is abnormal, it is also possible to consider resetting the offset back to 0. For example, if the system is data inconsistent, an attempt may be made to reset the offset back to 0 and resynchronize the data.

It should be noted that resetting the offset back to 0 will clear all existing data records, so care should be taken and important data backed up before doing so.

S3: when the system is used for external service for the first time, a user can send a data processing request to the application cluster app when carrying out business processing, and meanwhile, the data processing request comprises data updating information. This means that the user can send a data processing request to the application cluster with accompanying data update information when the system first serves the user.

After receiving the data processing request sent by the user, the application cluster app analyzes the data processing request and updates the database according to the data updating information. Meanwhile, the application cluster app also records the data processing request and the data updating information in a log form so as to facilitate subsequent audit and tracing.

When data processing is performed, the application cluster app can perform operations such as cleaning, sorting and analyzing on the data according to the requirements of the user so as to meet the requirements of the user. Meanwhile, the application cluster app can monitor the data processing process, and safety and stability in the data processing process are ensured.

After the data processing is completed, the application cluster app presents the results to the user in a visual manner so that the user can better understand the data and analyze the results. Meanwhile, the application cluster app also sends the processing result to the user in the form of a report so that the user can make subsequent decisions and analysis.

In a word, the application cluster app is used as one of core components of the system, and can meet the requirement of a user on data processing when the system is firstly used for external service and update the database according to data update information. Meanwhile, the application cluster app can monitor and present results for the data processing process so that the user can better understand and analyze the data.

S4: the load balancing server will distribute the request to an application middleware in the proxy's application cluster according to the load situation. This means that the load balancing server will distribute the requests to one of the application middleware in the agent's application cluster reasonably according to the load situation of the system.

When the load balancing server processes the request, the load balancing server can dynamically allocate according to the real-time system load condition. When the system load is light, the request is distributed to one application middleware in the application cluster; when the system is heavily loaded, the request is distributed to another application middleware in the application cluster. The policy of load balancing can effectively utilize system resources, and avoid the conditions of overload of the system or untimely processing of the request.

Meanwhile, the load balancing server also has a health checking function. The state of the application middleware can be checked regularly, and if one application middleware is found to be faulty or is too slow in response, the application middleware can be removed from the application cluster, so that the performance of the whole system is prevented from being influenced. The health check function can effectively ensure the stability and usability of the system.

In addition, the load balancing server also has a session maintaining function. When a user initiates a request, the load balancing server distributes the request to the same application middleware according to the information such as the IP address of the user so as to maintain the consistency of the session. The session maintaining function can improve user experience and avoid the situation that a user cannot maintain a state among a plurality of requests.

S5: after the application receives the request, the data processing request is processed, and the operation request database language is sent to the data operation middleware. This means that after an application receives a request, it will process these data processing requests and send the corresponding operation request database language to the data operation middleware.

The data manipulation middleware is a part responsible for interaction with the database, receives a data processing request from an application, and performs a corresponding database manipulation according to the requested content. This process includes querying, inserting, updating, deleting, etc. the data. The data operation middleware translates the operation requests into sentences which can be understood by the database and then sends the sentences to the database for execution.

After the application receives the requests, the data processing requests are first analyzed and processed. This process includes parsing, validating, and authorizing the request. After confirming that the request is legitimate and has sufficient rights, the application will send the corresponding operation request database language to the data operation middleware.

After receiving the operation requests, the data operation middleware generates corresponding SQL sentences according to the types and the contents of the requests. This process may involve querying, screening, aggregating, etc. operations on the data. After the SQL statement is generated, the data operation middleware sends the SQL statement to the database for execution.

After receiving the SQL statement, the database analyzes and executes the SQL statement. This process includes operations such as reading, writing, updating, etc. of the data. After execution is complete, the database returns the results to the data manipulation middleware. The data manipulation middleware processes and transforms the results and then returns the final results to the application.

S6: the data middleware can acquire the latest offset of the current table according to the operation service table and send an execution database statement to the main database of the service library. And after the operation SQL plus the offset is successful, storing the operation SQL plus the offset into a database server, and simultaneously updating the current offset information of the corresponding service library and the service table operated in the cache. At this time, the offsets of the master libraries A and B will be different, and the latest offset is denoted by S. The offset of the T1 table in the A library is the latest S, and the offset of the T1 table in the B library is S-1, and the storage mode is as follows: { key: t1, value: [ A: s, B: S-1] }. In order to ensure that the service data in the cache is the latest data, the cache information of the relevant table in the cache cluster should be deleted at the end of the operation.

The data middleware may also take into account some other factors in performing the operations described above. For example, if the data volume of the business table is large, frequently sending SQL statements to the master database may affect performance. Therefore, the data middleware also needs to properly adjust the acquisition offset and the frequency of sending SQL sentences according to the actual situation of the service table.

In addition, in order to better ensure the consistency of the data, the data middleware also needs to check whether the operation is successful after the SQL statement is executed. If an operation fails, it needs to be re-performed until it is successful. Meanwhile, in order to prevent the situation that the offset amount collides or is repeated, the data middleware also needs to carry out uniqueness check on the offset amount.

In addition, the data middleware needs to consider the expiration time of the cache. If the data in the cache has expired, then the most up-to-date data needs to be retrieved from the master database. Meanwhile, in order to reduce the access times to the main database, some cache strategies, such as a cache elimination strategy, a cache update strategy and the like, can be adopted by the data middleware.

S7: the data synchronization middleware should scan SQL log files of a plurality of database servers in the database cluster at regular time, and acquire the current synchronization offset of the target database table after acquiring the synchronization information. If the target library synchronization offset tc+1=td (Tc represents the current target table offset, td represents the intra-log operation offset), the current database operation statement is executed to the target library while the target library and target table offsets in the cache are updated. Otherwise, the message middleware is added to wait for the next execution. This means that the data synchronization middleware will periodically scan the SQL log files of multiple database servers, acquire the current synchronization offset of the target database table after acquiring the synchronization information. If the synchronization offset of the target library is consistent with the expected (i.e., tc+1=td), then the corresponding database operation statement is executed to the target library and the offsets of the target library and target table in the cache are updated. If the synchronization offset of the target library is inconsistent with the expected one, the operation is added to the message middleware to wait for the next execution.

The data synchronization middleware may also take into account a number of other factors in performing the operations described above. For example, if the target library table current offset is large, it may be necessary to wait for a period of time before performing the operation to avoid data collision. In addition, if the target library table already has the same operation, the operation can be ignored, avoiding repeated execution.

In implementing the data synchronization middleware, the following aspects need to be considered:

1. database connection management: middleware needs to connect multiple database servers and manage the establishment and disconnection of these connections.

2. Journal file parsing: the middleware needs to analyze the SQL log file and extract the related operation information.

3. Offset management: the middleware needs to manage the target library table and the offset of the target library, so as to ensure the correctness of data synchronization.

4. Message middleware integration: the middleware needs to add operations that cannot be performed immediately to the message middleware in order to be performed again when the target library is available.

5. Exception handling: middleware needs to handle various abnormal situations such as connection failure, parsing error, etc.

S8: the data synchronization middleware should monitor the message middleware at the same time, and execute the standard same step s7. When the message is repeatedly executed more than three times, if the pushing is not successful yet, the error information is pushed into the buffering middleware. This means that the data synchronization middleware will not only periodically scan the SQL log files of multiple database servers to obtain synchronization information, but also will monitor the message middleware. When the message is repeatedly executed for more than three times and is not successfully pushed, the error information is pushed into the buffer middleware for processing.

The implementation of the data synchronization middleware needs to monitor the message middleware at the same time, and the execution standard is the same as that of step S7. When the message is repeatedly executed for more than three times and is not successfully pushed, the error information is pushed into the buffer middleware for processing. In the repeated execution process of the message, if an abnormal situation occurs, error information needs to be recorded, and the error information is sent to other components through message middleware. In addition, the data synchronization middleware also needs to support scanning and parsing of SQL log files of multiple database servers to obtain synchronization information. In the data synchronization process, the integrity and consistency of the data need to be ensured, and the situation of data loss or repetition is avoided.

By simultaneously monitoring the message middleware and scanning the SQL log file, the data synchronization middleware can effectively realize the synchronization and sharing of data. When the message is repeatedly executed for more than three times and is still not successfully pushed, the error information is pushed into the buffer middleware, so that the abnormal situation in the message pushing process can be effectively solved, and the stability and reliability of data synchronization are ensured.

S9: the listening service should scan the cache for error information in real time. When the message middleware has error information, the information is pushed to related management personnel in time. The manager should check the existence of the problem and solve the problem of synchronization abnormality in time. This means that a listening service will scan the cache for error messages in real time, and once the message middleware is found to have error messages, it will push the messages to the relevant manager in time. Management personnel need to check the problem and solve the problem of synchronization abnormality in time.

S10: the user's query request to the business system should first retrieve the stored information from the cache. If this information is not present in the cache, a query sql should be sent to the database middleware. The database middleware splits the query target table and compares the offset queue of the corresponding table in the cache with the current offset. And if the offset amounts are consistent, indicating that the content of the database is the latest information. And sending the query request to a corresponding database, acquiring data and storing the data into a cache cluster. If not, the query cycle is entered. If the latest information is not acquired within 3 seconds, the synchronization abnormality is indicated. And fusing the current application and returning to the synchronous exception through a fusing mechanism.

S11: after the application receives the exception, the update to the database and the query request should be blocked, only the query request from the cache is processed and the synchronization status of the database is checked at regular time. Once the query synchronization is successful, the application should re-receive and respond to relevant requests involving database operations.

In addition to the above steps, this embodiment may further include the following operation steps:

s12: the main database in the database cluster should send SQL log files to the data synchronization middleware at regular time for the data synchronization operation. The data synchronization middleware should acquire the latest SQL log file from the main database and compare it with the SQL log of the target database to determine whether the latest operation exists. If the operation offset in the SQL log file of the main database is greater than the operation offset of the target database, the data in the main database is newer than the target database, and the data in the main database needs to be synchronized into the target database.

S13: when the data synchronization operation is executed, the data synchronization middleware should synchronize the data in the main database according to the order of the operation offset, and store the synchronized result into the target database. Meanwhile, the data synchronization middleware also needs to update offset information of a corresponding service library and a corresponding service table in the cache so as to ensure that the data in the cache is the latest data.

S14: if an abnormality occurs in the data synchronization process, the data synchronization middleware should immediately stop the synchronization operation and record error information. Meanwhile, error information should be pushed to the message middleware so that relevant management personnel can timely find and process the abnormality.

S15: in order to ensure the reliability of data synchronization, the primary database and the target database in the database cluster should perform data backup and recovery operations periodically. During the backup and restore process, backup software with high reliability should be used to operate to ensure the integrity and consistency of the data.

S16: in order to ensure the stability and reliability of the system, a plurality of database clusters should be deployed in different geographic locations and different network environments to realize disaster recovery and backup of data. Meanwhile, the backup data should be periodically checked and restored to ensure availability and integrity of the backup data.

S17: for business scenarios involving large amounts of data operations, the use of distributed database systems should be considered to improve the performance and reliability of the system. The distributed database system can disperse data to a plurality of database servers, and improves concurrency and response speed of the system. Meanwhile, the distributed database system can also provide data backup and recovery functions so as to prevent data loss and catastrophic failure.

S18: in a distributed database system, data should be scattered across multiple database servers using database sharding techniques. The database slicing technology can divide data into a plurality of slices according to service requirements and system load conditions, and store different slices on different database servers. This may improve the performance and reliability of the system and reduce the load on individual database servers.

S19: in a distributed database system, database replication techniques should be used to improve the performance and reliability of the system. Database replication techniques may replicate data onto multiple database servers to increase the availability and response speed of the system. Meanwhile, when a certain database server fails, the system can be quickly switched to other servers, and the availability of the system is ensured.

S20: in a distributed database system, database clustering techniques should be used to improve the performance and reliability of the system. Database clustering techniques may group multiple database servers into a cluster that collectively provide data services. This can improve concurrency and response speed of the system, and improve usability of the system. Meanwhile, the database cluster technology can also provide load balancing and fault tolerance functions so as to prevent single-point faults and overload problems.

According to the embodiment, when the single database is abnormal, no manual intervention is needed, services can still be normally provided for the outside, and the fault tolerance of the platform is greatly improved. This embodiment has the advantage that anomalies in the operation of the system are fully taken into account and an automated solution is devised to maintain the stability and usability of the system. Through an early warning mechanism, synchronization abnormality can be timely pushed to related personnel, so that the related personnel can timely find problems and take corresponding measures. The application of the early warning mechanism greatly enhances the reliability and safety of the system. In addition, the scheme has compatibility, can be compatible with various databases, and does not depend on the databases to provide a synchronization function. This allows the solution to the synchronization problem of different types of databases, some of which do not provide a synchronization solution, thus expanding their application range and applicability.

In order to ensure data consistency, the operation result needs to be stored in the message middleware after each writing operation is executed in the database. Each time the result of the operation is read in the message middleware, a corresponding write operation needs to be performed in the database.

In order to ensure data security, the operation result needs to be stored in the message middleware each time before the writing operation is executed in the database. Each time the result of the operation is read in the message middleware, a corresponding write operation needs to be performed in the database. If the results of the two operations are inconsistent, the current write operation should be stopped immediately and exception information returned.

To ensure data integrity, the operation results need to be stored in the message middleware each time a write operation is performed in the database. Each time the result of the operation is read in the message middleware, a corresponding write operation needs to be performed in the database. If the results of the two operations are inconsistent, the current write operation should be stopped immediately and exception information returned. After exception handling is complete, the write operation should be performed again to ensure data integrity.

In order to ensure data reliability, each time a write operation is performed in the database, the operation result needs to be stored in the message middleware. Each time the result of the operation is read in the message middleware, a corresponding write operation needs to be performed in the database. If the results of the two operations are inconsistent, the current write operation should be stopped immediately and exception information returned. After the exception handling is completed, the write operation should be performed again to ensure data reliability.

To ensure data availability, the results of the operations need to be stored in the message middleware each time a write operation is performed in the database. Each time the result of the operation is read in the message middleware, a corresponding write operation needs to be performed in the database. If the results of the two operations are inconsistent, the current write operation should be stopped immediately and exception information returned. After the exception handling is completed, the write operation should be performed again to ensure data availability.

The advantage of this particular embodiment is that anomalies in the operation of the system are fully accounted for and an automated solution is devised to maintain system stability and availability. Through an early warning mechanism, synchronization abnormality can be timely pushed to related personnel, so that the related personnel can timely find problems and take corresponding measures. The application of the early warning mechanism greatly enhances the reliability and safety of the system. In addition, the scheme has compatibility, can be compatible with various databases, and does not depend on the databases to provide a synchronization function. This allows the solution to the synchronization problem of different types of databases, some of which do not provide a synchronization solution, thus expanding their application range and applicability.

Of course, it is to be understood that other variations of the above detailed procedures are also possible, and all related variations should fall within the protection scope of the present invention.

On the basis of the above embodiments, another embodiment of the present invention provides another data updating method. The method comprises the following steps:

1. and receiving a data processing request of a user and generating a data query statement.

2. A target database is determined that executes the data query statement.

3. It is checked whether the current offset of the target database is the same as the offsets of the other master databases. If it is different, the following steps are performed; otherwise, go on to step 4.

4. And acquiring log files of other main databases according to the current offset of the target database and offset information of other main databases.

5. And updating the data of the target database according to the log file, and updating the current offset of the target database.

6. And repeatedly executing the steps 3-5 until the current offset of the target database is the same as the offsets of other main databases.

8. And sending out data query information.

According to the data updating method, the data updating of the target database is automatically realized by acquiring the offset information of the target database and other main databases and combining the log files of the other main databases, the updating process of the target database is not needed manually, the timeliness of the database updating is improved, the data processing request can be normally processed without manual intervention immediately when the database is abnormal, and the fault tolerance rate of the data request processing is greatly improved. Meanwhile, the problems of database error update and delayed update which are caused by unavoidable manual operation are avoided, and the efficiency and the accuracy of database update are improved. Furthermore, timeliness and efficiency of data processing requests are improved.

In another embodiment, the method may be further optimized. In particular, database synchronization may be performed faster by introducing a "snapshot" mechanism. In this mechanism, each master database creates a "snapshot" of the current database state on a regular basis (e.g., once per day). These "snapshots" may be stored in another separate database, referred to as a "snapshot database". When the target database needs to be updated, the latest snapshot can be obtained from the snapshot database, and then the data is updated according to the latest snapshot. The method has the advantages that the synchronization of a large amount of data can be completed in a short time, and the integrity and consistency of the data can be ensured.

In addition, an incremental update mechanism can be introduced to further improve the efficiency of data update. In this mechanism, each master database will create not only a "snapshot" of the full amount of data, but also incremental update records between different data tables. These incremental update records may be stored in another separate database, referred to as an "incremental update database". When the target database needs to be updated, the latest incremental update records can be acquired from the incremental update database, and then the target database is updated in an incremental mode according to the records. This has the advantage that the updating of the target database can be done faster and the performance impact on the target database can be reduced.

In general, embodiments of the present invention provide an efficient, automated, reliable database synchronization method. By introducing a snapshot mechanism and an increment updating mechanism, database synchronization can be performed faster, and the efficiency and accuracy of data updating are improved. Meanwhile, the problems of error update and delay update in manual operation are avoided, and the timeliness and efficiency of the data processing request are improved.

The embodiment of the invention also provides a data processing request execution device, as expressed in the following embodiment. Since the principle of the device for solving the problem is similar to that of the data processing request execution method, the implementation of the device can refer to the implementation of the data processing request execution method, and the repetition is omitted.

The embodiment of the invention also provides a device for executing the data processing request, which is used for improving the efficiency and the accuracy of updating the database and improving the timeliness and the efficiency of executing the data processing request, as shown in fig. 3, and the device comprises:

a data query statement generation module 301, configured to generate a data query statement based on a received data processing request of a user; determining a target database for executing the data query statement;

The offset determining module 302 is configured to determine a current offset of the target database from preset database offset information; when the current offset is different from the offsets of other main databases, acquiring offset information and log files of the other main databases; the database offset information comprises offset information of different main databases; the offset information is used for representing update information of the main database; the log file is used for updating data of different data tables in the main database;

a data updating module 303, configured to perform data updating on a target database according to the offset information and the log file of the other main databases, and update the current offset of the target database; repeating the steps until the current offset is the same as the offsets of other main databases;

a data query statement execution module 304, configured to execute the data query statement with the updated target database to obtain data query information; and sending out the data query information.

In one embodiment, the data query statement generation module is specifically configured to:

the offset determining module is specifically configured to:

the data updating module is specifically used for:

In one embodiment, as shown in fig. 4, further comprising:

an alarm module 401 for:

In one embodiment, the data query statement execution module 304 is specifically configured to: executing the data query statement with the updated target database to obtain data query information; sending the data query information to corresponding applications in an application cluster; the data query information is provided by the application to the corresponding user.

In one embodiment, the apparatus further comprises: the repeated execution module is used for repeatedly executing the steps until the current offset is the same as the offsets of other main databases; the repeated execution module is specifically configured to: and when the current offset is different from the offsets of other main databases, executing the steps again.

In one embodiment, the apparatus further comprises: the log recording module is used for recording the related log information of the data updating process and/or the data query process; the log recording module is specifically used for: and recording information such as time of data updating of the target database, executed SQL sentences, updated data quantity and the like.

In one embodiment, the apparatus further comprises: the abnormality detection module is used for sending alarm information representing the occurrence of abnormality when the abnormality occurs in the data updating process and/or the data query process; the abnormality detection module is specifically configured to: detecting whether an abnormality occurs in the data updating process and/or the data query process, such as SQL statement execution error, database connection failure, etc.; when an abnormality occurs, alarm information representing the occurrence of the abnormality is sent out.

By the device embodiment, the efficiency and the accuracy of database updating are improved, and the timeliness and the efficiency of executing data processing requests are also improved. Meanwhile, by the cooperation of the repeated execution module, the log recording module, the abnormality detection module, the fault diagnosis module, the early warning module, the self-maintenance module and other devices, the stability and the reliability of the execution of the data processing request and the updating process of the database can be better ensured.

An embodiment of the present invention provides a computer device for implementing all or part of the content in the above data processing request execution method, where the computer device specifically includes the following contents:

a processor (processor), a memory (memory), a communication interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete communication with each other through the bus; the communication interface is used for realizing information transmission between related devices; the computer device may be a desktop computer, a tablet computer, a mobile terminal, or the like, and the embodiment is not limited thereto. In this embodiment, the computer device may be implemented with reference to an embodiment for implementing a method for implementing a data processing request and an embodiment for implementing a device for implementing a data processing request, and the contents thereof are incorporated herein, and are not repeated here.

Fig. 5 is a schematic block diagram of a system configuration of a computer device 1000 according to an embodiment of the present application. As shown in fig. 5, the computer device 1000 may include a central processor 1001 and a memory 1002; the memory 1002 is coupled to the central processor 1001. Notably, this fig. 5 is exemplary; other types of structures may also be used in addition to or in place of the structures to implement telecommunications functions or other functions.

In one embodiment, the data processing request execution function may be integrated into the central processor 1001. The central processor 1001 may be configured to control, among other things, the following:

In another embodiment, the data processing request execution apparatus may be configured separately from the central processor 1001, for example, the data processing request execution apparatus may be configured as a chip connected to the central processor 1001, and the data processing request execution function is realized by control of the central processor.

As shown in fig. 5, the computer device 1000 may further include: a communication module 1003, an input unit 1004, an audio processor 1005, a display 1006, a power supply 1007. It is noted that the computer device 1000 need not include all of the components shown in FIG. 5; in addition, the computer device 1000 may further include components not shown in fig. 5, to which reference is made to the prior art.

As shown in fig. 5, the central processor 1001, sometimes also referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, and the central processor 1001 receives input and controls the operation of the various components of the computer device 1000.

The memory 1002 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information about failure may be stored, and a program for executing the information may be stored. And the central processor 1001 can execute the program stored in the memory 1002 to realize information storage or processing, and the like.

The input unit 1004 provides input to the central processor 1001. The input unit 1004 is, for example, a key or a touch input device. The power supply 1007 is used to provide power to the computer device 1000. The display 1006 is used for displaying display objects such as images and characters. The display may be, for example, but not limited to, an LCD display.

The memory 1002 may be a solid state memory such as Read Only Memory (ROM), random Access Memory (RAM), SIM card, and the like. But also a memory which holds information even when powered down, can be selectively erased and provided with further data, an example of which is sometimes referred to as EPROM or the like. Memory 1002 may also be some other type of device. Memory 1002 includes a buffer memory 1021 (sometimes referred to as a buffer). The memory 1002 may include an application/function storage 1022, the application/function storage 1022 for storing application programs and function programs or for executing a flow of operations of the computer apparatus 1000 by the central processor 1001.

The memory 1002 may also include a data store 1023, the data store 1023 for storing data such as contacts, digital data, pictures, sounds, and/or any other data used by a computer device. The driver store 1024 of the memory 1002 can include various drivers for the computer device for communication functions and/or for performing other functions of the computer device (e.g., messaging applications, address book applications, etc.).

The communication module 1003 is a transmitter/receiver 1003 that transmits and receives signals via an antenna 1008. A communication module (transmitter/receiver) 1003 is coupled to the central processor 1001 to provide an input signal and receive an output signal, which may be the same as in the case of a conventional mobile communication terminal.

Based on different communication technologies, a plurality of communication modules 1003, such as a cellular network module, a bluetooth module, and/or a wireless lan module, etc., may be provided in the same computer device. The communication module (transmitter/receiver) 1003 is also coupled to a speaker 1009 and a microphone 1010 via an audio processor 1005 to provide audio output via the speaker 1009 and to receive audio input from the microphone 1010 to implement usual telecommunications functionality. The audio processor 1005 may include any suitable buffers, decoders, amplifiers and so forth. In addition, an audio processor 1005 is also coupled to the central processor 1001 so that sound can be recorded locally through the microphone 1010 and so that sound stored locally can be played through the speaker 1009.

It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The foregoing description of the embodiments has been provided for the purpose of illustrating the general principles of the invention, and is not meant to limit the scope of the invention, but to limit the invention to the particular embodiments, and any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the invention are intended to be included within the scope of the invention.

Claims

1. A method of executing a data processing request, comprising:

2. The method of claim 1, wherein generating a data query statement based on the received data processing request of the user comprises:

3. The method of claim 1, wherein the data query statement corresponds to a target data table in a target database;

4. The method as recited in claim 1, further comprising:

5. A data processing request execution apparatus, comprising:

6. The apparatus of claim 5, wherein the data query statement generation module is specifically configured to:

7. The apparatus of claim 5, wherein the data query statement corresponds to a target data table in a target database;

The offset determining module is specifically configured to:

the data updating module is specifically used for:

8. The apparatus as recited in claim 5, further comprising:

an alarm module for:

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 1 to 4 when executing the computer program.

10. A computer readable storage medium, characterized in that the computer readable storage medium stores a computer program which, when executed by a processor, implements the method of any of claims 1 to 4.

11. A computer program product, characterized in that it comprises a computer program which, when executed by a processor, implements the method of any of claims 1 to 4.