CN115964441A - System for consistency is write more to database - Google Patents
System for consistency is write more to database Download PDFInfo
- Publication number
- CN115964441A CN115964441A CN202211716514.8A CN202211716514A CN115964441A CN 115964441 A CN115964441 A CN 115964441A CN 202211716514 A CN202211716514 A CN 202211716514A CN 115964441 A CN115964441 A CN 115964441A
- Authority
- CN
- China
- Prior art keywords
- middleware
- transaction identifier
- service request
- databases
- database
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a system for multi-write consistency of a database. The system comprises: the middleware and the transaction identifier generation module; the middleware is used for sending a transaction identifier generation request to the transaction identifier generation module when receiving a service request sent by an application end, forwarding the service request to a database, and sending a transaction identifier release request to the transaction identifier generation module when receiving information of successful processing of the service request returned by the database; the transaction identifier generation module is used for generating a corresponding transaction identifier when receiving a transaction identifier generation request sent by the middleware; and when a transaction identifier release request sent by the middleware is received, releasing the transaction identifier. According to the scheme provided by the invention, when the service side needs to carry out service requests, the middleware can be used for simultaneously executing service operation on the plurality of databases, so that the data consistency in the plurality of databases is ensured.
Description
Technical Field
The invention relates to the technical field of databases, in particular to a system for multi-write consistency of a database.
Background
In the existing common architecture for disaster recovery of the main database and the standby database, the main database generally provides services to the outside, and the standby database is consistent with the main database in a data synchronization manner. In the method, in order to ensure the consistency of the primary data copy and the secondary data copy, the performance of a primary library needs to be sacrificed; or in order to ensure that the main library can respond to the service request in time, and the synchronous response of the standby libraries is not waited, thereby sacrificing the defect of the consistency of the copies. When the primary database responds differently, hidden danger of inconsistent data of the primary and secondary databases is brought, so that failure of switching of the primary and secondary databases is caused. Meanwhile, in the process of switching the main database and the standby database, the service generates a failed request, and the continuity of the service cannot be ensured.
Disclosure of Invention
In order to solve the technical problem that a plurality of databases cannot ensure data consistency when a service side, namely an application side, performs write operation on the plurality of databases at the same time, embodiments of the present invention provide a system for database multi-write consistency.
The technical scheme of the embodiment of the invention is realized as follows:
the embodiment of the invention provides a system for multi-write consistency of a database, which comprises: the middleware and the transaction identifier generation module;
the middleware is used for sending a transaction identifier generation request to the transaction identifier generation module when receiving a service request sent by an application end, forwarding the service request to a database, and sending a transaction identifier release request to the transaction identifier generation module when receiving information of successful processing of the service request returned by the database;
the transaction identifier generation module is used for generating a corresponding transaction identifier when receiving a transaction identifier generation request sent by the middleware; and when a transaction identifier release request sent by the middleware is received, releasing the transaction identifier.
In an embodiment, the middleware is configured to forward the service request to multiple databases, and send a transaction identifier release request to the transaction identifier generation module when receiving information that all the databases return successful service request processing.
In an embodiment, the middleware is further configured to send the information that the service request processing is successful to the application side when receiving the information that all the databases return the service request processing is successful.
In an embodiment, the middleware is further configured to send the information that the service request processing is successful to the application side when the information that the configuration required number of databases all return the service request processing success is received.
In an embodiment, the middleware is further configured to generate a reverse operation instruction with a transaction identifier when processing of a part of the database service requests is successful and processing of a part of the database service requests is failed, and issue the reverse operation instruction to all databases, so that all databases execute reverse rollback operation.
In an embodiment, the middleware is further configured to send the backward operation instruction to the transaction identifier generating module for storage, and send a transaction identifier release request to the transaction identifier generating module.
In an embodiment, the middleware is further configured to send information of failure in processing the service request to the application terminal when receiving information of successful execution of the reverse operation instruction returned by all the databases.
In an embodiment, when there are a plurality of middleware, the middleware is further configured to, when abnormality occurs after other middleware generates a reverse operation instruction, obtain the reverse operation instruction generated by other middleware from the transaction identifier generation module, and issue the reverse operation instruction to all databases, so that all databases execute reverse rollback operation.
In an embodiment, when there are a plurality of middleware, the middleware is further configured to, when abnormality occurs after other middleware forwards a service request to all databases, obtain, from the transaction identifier generation module, a transaction identifier that has been generated by the other middleware and has not been released, and check whether all databases are successfully processed with the service request according to the transaction identifier.
In an embodiment, the middleware is further configured to send information that service request processing is successful to the application terminal when all database service request processing is verified to be successful according to the transaction identifier; and when the partial database service request is successfully processed and the partial database service request is unsuccessfully processed according to the transaction identifier, generating a reverse operation instruction with the transaction identifier, and issuing the reverse operation instruction to all databases to enable all databases to execute reverse rollback operation.
The system for the multi-write consistency of the database provided by the embodiment of the invention comprises a middleware and a transaction identifier generation module; the middleware is used for sending a transaction identifier generation request to the transaction identifier generation module when receiving a service request sent by an application end, forwarding the service request to a database, and sending a transaction identifier release request to the transaction identifier generation module when receiving information of successful processing of the service request returned by the database; the transaction identifier generation module is used for generating a corresponding transaction identifier when receiving a transaction identifier generation request sent by the middleware; and when a transaction identifier release request sent by the middleware is received, releasing the transaction identifier. According to the scheme provided by the invention, when the service side needs to carry out service requests, the middleware can be used for simultaneously executing service operation on the plurality of databases, so that the data consistency in the plurality of databases is ensured.
Drawings
FIG. 1 is a schematic structural diagram of a system for database multi-write consistency according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a data copy peer-to-peer disaster recovery system according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a transaction flow according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples.
The embodiment of the invention provides a database multi-write consistency system, as shown in fig. 1, the system includes a middleware 101 and a transaction identifier generation module 102;
the middleware 101 is configured to send a transaction identifier generation request to the transaction identifier generation module when receiving a service request sent by an application, forward the service request to a database, and send a transaction identifier release request to the transaction identifier generation module when receiving information that the service request returned by the database is successfully processed;
the transaction identifier generating module 102 is configured to generate a corresponding transaction identifier when receiving a transaction identifier generating request sent by the middleware; and when a transaction identifier release request sent by the middleware is received, releasing the transaction identifier.
In this embodiment, a set of stateless middleware 101 (not storing requestor information) and a set of modules that can generate monotonically increasing transaction identifiers, that is, a transaction identifier generation module 102, may be specifically designed between the business application end and the database, and the middleware 101 reads and writes all copies of the database in parallel, interacts with the transaction identifier generation module 102, and ensures data consistency in all databases when multiple writes are performed to the databases simultaneously.
Here, the application end can be used to send service request to the middleware, the database, can be used to receive and process the service request forwarded by the middleware,
the embodiment can ensure the consistency of all data copies under the high availability and disaster recovery architecture of the database.
In an embodiment, the middleware is further configured to forward the service request to multiple databases, and send a transaction identifier release request to the transaction identifier generation module when receiving information that all the databases return that the service request is successfully processed.
In the embodiment, the application end executes write operation on a plurality of databases by using the middleware at the same time, so that the technical problems of service lack of continuity and performance reduction caused by switching between the main database and the standby database when an individual database is abnormal are solved.
The embodiment writes a plurality of databases in parallel based on the middleware, and can ensure the consistency of all data copies.
Referring now to fig. 2, a detailed example will be presented to illustrate the present embodiment.
In fig. 2, an application side sends a service request to a middleware (middleware 1 to middleware N) through a switch, the middleware is connected with a transaction ID generation module, i.e., a transaction identifier generation module (the transaction ID generation module is divided into a main transaction ID generation module and a standby transaction ID generation module), and the middleware forwards the service request to a database (database 1 to database N).
In the system, all databases are peer-to-peer identities, and the main database and the standby database are not distinguished.
When the service side, namely the application side, makes a write request, the service request is firstly sent to the middleware, and the middleware makes write requests for all databases. Before making a write request, the middleware requests an ID from the transaction ID generation module as the transaction identification of the write request.
If the middleware receives the successful writing information of all (or the quantity required by configuration) databases, the middleware informs the transaction ID generation module of the release of the transaction ID identification applied by the current transaction, and the service request is successfully rewritten.
If the partial database write succeeds and the partial database write fails, the middleware generates a reverse operation SQL (structured query language) with the transaction ID corresponding to the transaction, namely a reverse operation instruction, issues the reverse operation SQL to all databases, executes reverse rollback operation, simultaneously sends the reverse operation SQL to the transaction ID generation module for storage, and simultaneously informs the transaction ID generation module of the release of the transaction ID identifier applied by the current transaction. And when the middleware receives the execution success messages returned by all the databases needing to execute the reverse operation SQL and the transaction ID generation module receives the messages of the reverse operation SQL, the middleware replies a write failure to the service.
Considering that a part of databases need to execute reverse rollback SQL due to write failure, waiting time delay caused by drawing down a normal database due to consistency requirements is reduced, or the part of databases are down to cause that the normal databases cannot respond back, the middleware can configure the number of messages returned by the successful writing databases, and when the configured number of messages is met, the success of the service can be returned.
The transaction ID generation module provides an application transaction ID, a release transaction ID, a query applied unreleased ID, a storage write transaction reverse rollback SQL 4 interface for the middleware, and records the corresponding relation between the middleware and the applied transaction ID in a file mode. And recording the files of the corresponding relation, and transmitting the files among the plurality of transaction ID generation modules in real time. When the transaction ID generation module receives a request of applying for a transaction ID by the middleware, generating an 8-byte monotonically increasing ID, returning the ID to the middleware, locally recording the ID, a transaction state and a corresponding middleware number, wherein a transaction flag bit is empty by default and represents that the transaction ID is in a use state; when a request for the release of the transaction ID is received by the middleware, the ID is found in the local record, and the transaction flag bit is recorded as 1, which represents that the transaction ID is invalid. When the affair ID generation module receives a request for requesting for unreleased ID by the middleware application query, the ID list with a null flag bit is searched in the local file according to the middleware number in the request and returned to the middle. When the transaction ID generation module receives the write transaction reverse rollback SQL sent by the middleware, the middleware number and the corresponding SQL are stored locally for subsequent inquiry according to the middleware number.
In this embodiment, because the processing logic for applying for the transaction ID and releasing the transaction ID is simple, and the requirement on the network bandwidth is low, higher concurrency and lower response delay can be supported, and therefore, the influence on the execution efficiency of the transaction is limited. Meanwhile, when the service request initiates a read operation, the middleware randomly issues the read request to any one data copy. Therefore, under the condition of high concurrency, the concurrency of the read transaction can be improved in multiples without special business processing, and the performance of the read transaction is integrally improved; in addition, the middleware is designed in a stateless mode, more than 2 middleware can be deployed, and single-node fault risks are avoided in a redundant mode. The number of the transaction ID generation modules can be more than 2, the master module sends the added information to the standby module in a network mode at the colleagues writing the local documents, and the single-node fault risk is avoided.
In this embodiment, the service request may be delivered to any middleware in a soft and hard load balancing manner such as F5, loadbalance, nginx, LVS, and the like.
The database system provided by the embodiment of the invention comprises an application end, a middleware, a transaction identifier generation module and a database; the application terminal is used for sending a service request to the middleware; the middleware is used for sending a transaction identifier generation request to the transaction identifier generation module when receiving a service request sent by the application terminal; the service request is forwarded to a database, and when information that the service request returned by the database is successfully processed is received, a transaction identifier release request is sent to the transaction identifier generation module; the transaction identifier generation module is used for generating a corresponding transaction identifier when receiving a transaction identifier generation request sent by the middleware; when a transaction identifier release request sent by the middleware is received, releasing the transaction identifier; and the database is used for receiving and processing the service request forwarded by the middleware. The scheme provided by the invention can ensure the data consistency of a plurality of databases when the service side executes the service requests to the plurality of databases at the same time.
The present invention will be described in detail below with reference to application examples.
Specifically, the present embodiment provides a method and an apparatus for ensuring consistency of multiple writes to a database. Referring to fig. 3, for the write transaction flow, the present embodiment includes the following:
the service request issues a write transaction request (insert and update operation) to the middleware, and the middleware acquires an ID value from the transaction ID generation module and issues the request to all database nodes.
And after receiving the message that all (or the required quantity of) databases return success, the middleware executes a success message on the business write-back transaction.
The middleware receives the information of the execution failure of the partial database, or does not receive the response requests of the configuration required quantity within the stipulated time, which indicates that the execution of the write transaction of the partial database nodes fails, generates a reverse backspace SQL request according to the ID number of the write transaction, and sends the request together with the request for releasing the transaction ID to the transaction ID generation module for backup. Meanwhile, the middleware reversely backs the SQL request, issues the SQL request to the database node which succeeds in commit, and executes the revocation work of the submitted transaction.
And when the middleware receives the message that the transaction ID generation module responds and executes the reverse rollback SQL successfully, returning an execution failure message to the service.
Meanwhile, for the read transaction flow, the embodiment includes the following contents:
the service request issues a read transaction request to the middleware, and the middleware randomly issues the transaction request to any database node.
And after receiving the database reading request, the middleware forwards the database reading request to the service.
In addition, the processing flow when the middleware is abnormal includes the following contents:
when the exception of the middleware occurs when the service request is received but not sent to the database, no intervention is needed, and because the middleware is designed in a stateless manner, the service overtime retry request can be sent to the normal middleware.
When the exception of the middleware occurs in the successful operation of a partial database and the failed operation of the partial database, if the reverse backspace SQL is generated and is synchronized to the transaction ID generation module, the generated reverse backspace SQL is acquired from the transaction ID generation module in other middleware and is issued to all databases which are successfully executed, and the submitted transaction is rolled back; if the reverse rollback SQL is not generated, manual intervention is needed for rollback operation.
When the exception of the middleware occurs when the service request is received and sent to the database, but the database does not respond or does not respond completely, at this time, the transaction ID acquired but not released by the exception middleware is acquired from the transaction ID production module on other middleware, and whether all databases are successfully executed is checked according to the transaction ID.
The method provided by the patent has the following beneficial effects:
1. the risk of failure of main-standby switching in the conventional database main-standby disaster tolerance architecture can be avoided, and the performance of the read transaction of the database under the multi-database replica disaster tolerance scene is improved.
2. Because the method has no invasion to the database, double writing or multiple writing can be carried out on different databases, and the method can be used for realizing the purpose of disaster tolerance and seamless replacement of different databases.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A system for database write-many consistency, the system comprising: the middleware and the transaction identifier generation module;
the middleware is used for sending a transaction identifier generation request to the transaction identifier generation module when receiving a service request sent by an application end, forwarding the service request to a database, and sending a transaction identifier release request to the transaction identifier generation module when receiving information of successful processing of the service request returned by the database;
the transaction identifier generation module is used for generating a corresponding transaction identifier when receiving a transaction identifier generation request sent by the middleware; and when a transaction identifier release request sent by the middleware is received, releasing the transaction identifier.
2. The system according to claim 1, wherein the middleware is configured to forward the service request to multiple databases, and send a transaction identifier release request to the transaction identifier generation module when receiving information that all databases return successful service request processing.
3. The system according to claim 2, wherein the middleware is further configured to send the information that the service request processing is successful to the application side when receiving the information that all the databases return the service request processing is successful.
4. The system according to claim 2, wherein the middleware is further configured to send the information that the service request processing is successful to the application side when the information that the configuration required number of databases all return the information that the service request processing is successful is received.
5. The system according to claim 2, wherein the middleware is further configured to generate a reverse operation instruction with a transaction identifier when the partial database service request processing is successful and the partial database service request processing is failed, and issue the reverse operation instruction to all databases to enable all databases to perform reverse rollback operation.
6. The system according to claim 5, wherein the middleware is further configured to send the backward operation instruction to the transaction identifier generation module for saving, and send a transaction identifier release request to the transaction identifier generation module.
7. The system according to claim 6, wherein the middleware is further configured to send information of failure in processing the service request to the application terminal when receiving information that the reverse operation instruction returned by all databases is successfully executed.
8. The system according to claim 7, wherein when there are a plurality of middleware, the middleware is further configured to, when an exception occurs after other middleware generates a reverse operation instruction, obtain a reverse operation instruction generated by other middleware from the transaction identifier generation module, and issue the reverse operation instruction to all databases, so that all databases execute reverse fallback operation.
9. The system according to claim 7, wherein when there are a plurality of middleware, the middleware is further configured to, when an exception occurs after other middleware forwards a service request to all databases, obtain, from the transaction identifier generation module, a transaction identifier that has been generated by the other middleware and has not been released, and check whether all databases are successfully processed with the service request according to the transaction identifier.
10. The system according to claim 9, wherein the middleware is further configured to send information that the service request processing is successful to the application terminal when all database service request processing is verified to be successful according to the transaction identifier; and when the partial database service request is successfully processed and the partial database service request is unsuccessfully processed according to the transaction identifier, generating a reverse operation instruction with the transaction identifier, and issuing the reverse operation instruction to all databases to enable all databases to execute reverse rollback operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211716514.8A CN115964441A (en) | 2022-12-29 | 2022-12-29 | System for consistency is write more to database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211716514.8A CN115964441A (en) | 2022-12-29 | 2022-12-29 | System for consistency is write more to database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115964441A true CN115964441A (en) | 2023-04-14 |
Family
ID=87358398
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211716514.8A Pending CN115964441A (en) | 2022-12-29 | 2022-12-29 | System for consistency is write more to database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115964441A (en) |
-
2022
- 2022-12-29 CN CN202211716514.8A patent/CN115964441A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3435604B1 (en) | Service processing method, device, and system | |
| US11614867B2 (en) | Distributed storage system-based data processing method and storage device | |
| CN113239013B (en) | Distributed system and storage medium | |
| JP5264077B2 (en) | Geographically distributed cluster | |
| CN106776130B (en) | Log recovery method, storage device and storage node | |
| US5657440A (en) | Asynchronous remote data copying using subsystem to subsystem communication | |
| US6823474B2 (en) | Method and system for providing cluster replicated checkpoint services | |
| US20120036394A1 (en) | Data recovery method, data node, and distributed file system | |
| US20060080574A1 (en) | Redundant data storage reconfiguration | |
| KR20020010324A (en) | Transaction Management Method For Data Synchronous In Dual System Environment | |
| CN105574187A (en) | Duplication transaction consistency guaranteeing method and system for heterogeneous databases | |
| CN112130758B (en) | Data reading request processing method and system, electronic equipment and storage medium | |
| US9396076B2 (en) | Centralized version control system having high availability | |
| CN103544081B (en) | The management method of double base data server and device | |
| WO2022048416A1 (en) | Operation request processing method and apparatus, and device, and readable storage medium, and system | |
| WO2021082925A1 (en) | Transaction processing method and apparatus | |
| WO2023151443A1 (en) | Synchronizing main database and standby database | |
| CN111752892B (en) | Distributed file system and implementation method, management system, equipment and medium thereof | |
| CN115964441A (en) | System for consistency is write more to database | |
| CN107045426B (en) | Multi-copy reading method and system | |
| CN115658245A (en) | Transaction submitting system, method and device based on distributed database system | |
| US20210240351A1 (en) | Remote copy system and remote copy management method | |
| US7933873B2 (en) | Handling transfer of bad data to database partitions in restartable environments | |
| KR20090041624A (en) | Method and apparatus for synchronizing data in active-active database server system | |
| KR20130043823A (en) | Distributed storage system for maintaining data consistency based on log, and method for the same |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |