CN117667319A - Transaction processing method and device - Google Patents
Transaction processing method and device Download PDFInfo
- Publication number
- CN117667319A CN117667319A CN202410147781.0A CN202410147781A CN117667319A CN 117667319 A CN117667319 A CN 117667319A CN 202410147781 A CN202410147781 A CN 202410147781A CN 117667319 A CN117667319 A CN 117667319A
- Authority
- CN
- China
- Prior art keywords
- transaction
- data
- processing
- sub
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title abstract description 10
- 238000012545 processing Methods 0.000 claims abstract description 227
- 238000000034 method Methods 0.000 claims abstract description 58
- 230000004044 response Effects 0.000 claims abstract description 29
- 230000008569 process Effects 0.000 claims abstract description 21
- 230000007246 mechanism Effects 0.000 claims abstract description 16
- 238000004590 computer program Methods 0.000 claims description 16
- 238000004458 analytical method Methods 0.000 claims description 10
- 238000005516 engineering process Methods 0.000 claims description 9
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 238000004140 cleaning Methods 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000005096 rolling process Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 15
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000015556 catabolic process Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a transaction processing method and device, and relates to the technical field of data processing. One embodiment of the method comprises the following steps: receiving a transaction which is transmitted by a client and aims at a target database; determining a database type of a target database, reading pre-warning conditions configured for the database type in advance, and acquiring a pre-warning quantity threshold value from the pre-warning conditions; determining a transaction splitting strategy according to the transaction type to split the transaction into a plurality of sub-transactions in response to the data volume of the data set being greater than or equal to the early warning number threshold; and processing each sub-transaction by using a data processing mode to obtain a plurality of processing results, and returning a processing success message to the client in response to the success of the plurality of processing results. The embodiment is provided with a large transaction early warning mechanism, and can monitor the transaction in the batch processing process and early warn beyond a threshold value so as to automatically split the transaction by using a splitting strategy, thereby improving the processing efficiency and accuracy of batch transaction and reducing the burden brought to a database.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a transaction processing method and apparatus.
Background
When running batches for a system such as a banking system, handling large transactions for batch data is a common problem. When the amount of data to be processed in a transaction is large, for example, insert, update or delete operations are performed in large batches, a large pressure is caused on the database, so that performance is reduced and database error reporting problems occur.
To solve the above problem, large transactions are split mainly through business scenarios in the prior art: before development, determining a service scene, analyzing service logic including data flow direction, operation sequence, association relation and the like, controlling the data quantity generated by statement execution according to different service logic, and splitting the data quantity into different sub-transactions. And the threshold value of the transaction data size is increased under the condition of large transaction data size so as to avoid the conditions of error execution, database alarm, database performance breakdown and the like.
However, the existing mode needs to analyze business logic and re-write codes to solve the problem of transaction splitting, which increases the complexity and development difficulty of the system, and if the transaction splitting is excessively fine, the management and maintenance may be difficult.
Disclosure of Invention
In view of this, embodiments of the present invention provide a transaction processing method and apparatus, which at least can solve the problem in the prior art that code needs to be re-implemented to solve the complexity of splitting a transaction.
To achieve the above object, according to one aspect of an embodiment of the present invention, there is provided a transaction processing method including:
receiving a transaction which is transmitted by a client and aims at a target database; the transaction comprises a transaction type, a data set to be processed and a data processing mode;
determining a database type of a target database, reading pre-warning conditions configured for the database type in advance, and acquiring a pre-warning quantity threshold value from the pre-warning conditions;
determining a transaction splitting strategy according to the transaction type to split the transaction into a plurality of sub-transactions in response to the data volume of the data set being greater than or equal to the early warning volume threshold;
and processing each sub-transaction by using the data processing mode to obtain a plurality of processing results, and returning a processing success message to the client in response to the success of the plurality of processing results.
Optionally, the transaction type is a structured query SQL statement, and the receiving the transaction for the target database transmitted by the client includes:
receiving SQL sentences aiming at a target database and transmitted by a client, and using an SQL analysis library to analyze the SQL sentences in a grammar way so as to analyze the SQL sentences into grammar trees, and acquiring a data set to be processed and a data processing mode in the SQL sentences from the grammar trees;
The determining a transaction splitting policy according to the transaction type to split the transaction into a plurality of sub-transactions includes:
reading a metadata name from the transaction, determining a metadata table corresponding to the metadata name, determining a primary key of the metadata table, and acquiring a maximum value and a minimum value of the primary key;
splitting a section formed by the maximum value and the minimum value by using a preset section to obtain a plurality of sub-sections, and generating sub-transactions based on data corresponding to the sub-sections in the metadata table.
Optionally, the transaction type is data to be inserted, and the receiving the transaction for the target database transmitted by the client includes:
receiving data to be inserted aiming at a target database and a data processing mode transmitted by a client; the data is a file uploaded by the client, or an external interface is called to acquire a data set from a data acquisition path according to the data acquisition path transmitted by the client;
storing the data in the data set or the file into the temporary file row by row according to the data acquisition time sequence, and dividing the fields of each row by using a specific separator;
determining a transaction splitting policy according to the transaction type to split the transaction into a plurality of sub-transactions, including:
And acquiring the number of lines in the temporary file, and splitting the number of lines in the temporary file by using the preset number of lines to obtain a plurality of sub-transactions.
Optionally, the method further comprises: and performing data cleaning and format conversion processing on the data in the file or the data set.
Optionally, the data processing mode is parallel processing or serial processing, and the processing each sub-transaction using the data processing mode obtains a plurality of processing results, including:
in response to the data processing mode being parallel processing, utilizing a parallel computing technology to process a plurality of sub-transactions in parallel to obtain a plurality of processing results;
and responding to the data processing mode as serial processing, and sequentially processing each sub-transaction according to the splitting sequence among the sub-transactions to obtain a plurality of processing results.
Optionally, the method further comprises:
monitoring the processing result of each sub-transaction to call an exception handling mechanism to reprocess the target sub-transaction when the processing result of the monitored target sub-transaction is failure;
and in response to the fact that the processing result of the target sub-transaction is still failure under the condition that the preset retry times are reached, determining that the processing result is other sub-transactions which are successful, rolling back the data processed by the other sub-transactions aiming at the target database, and returning a processing failure message to the client.
To achieve the above object, according to another aspect of an embodiment of the present invention, there is provided a transaction processing apparatus including:
the data receiving module is used for receiving the transaction which is transmitted by the client and aims at the target database; the transaction comprises a transaction type, a data set to be processed and a data processing mode;
the early warning module is used for determining the database type of the target database, reading early warning conditions configured for the database type in advance, and acquiring early warning quantity threshold values from the early warning conditions;
the splitting module is used for determining a transaction splitting strategy according to the transaction type in response to the data volume of the data set being greater than or equal to the early warning number threshold value so as to split the transaction into a plurality of sub-transactions;
and the processing module is used for processing each sub-transaction by using the data processing mode to obtain a plurality of processing results, and returning a processing success message to the client in response to the success of the plurality of processing results.
Optionally, the transaction type is a structured query SQL statement, and the data receiving module is configured to:
receiving SQL sentences aiming at a target database and transmitted by a client, and using an SQL analysis library to analyze the SQL sentences in a grammar way so as to analyze the SQL sentences into grammar trees, and acquiring a data set to be processed and a data processing mode in the SQL sentences from the grammar trees;
The splitting module is used for:
reading a metadata name from the transaction, determining a metadata table corresponding to the metadata name, determining a primary key of the metadata table, and acquiring a maximum value and a minimum value of the primary key;
splitting a section formed by the maximum value and the minimum value by using a preset section to obtain a plurality of sub-sections, and generating sub-transactions based on data corresponding to the sub-sections in the metadata table.
Optionally, the transaction type is data to be inserted, and the data receiving module is configured to:
receiving data to be inserted aiming at a target database and a data processing mode transmitted by a client; the data is a file uploaded by the client, or an external interface is called to acquire a data set from a data acquisition path according to the data acquisition path transmitted by the client;
storing the data in the data set or the file into the temporary file row by row according to the data acquisition time sequence, and dividing the fields of each row by using a specific separator;
the splitting module is used for: and acquiring the number of lines in the temporary file, and splitting the number of lines in the temporary file by using the preset number of lines to obtain a plurality of sub-transactions.
Optionally, the data receiving module is further configured to:
And performing data cleaning and format conversion processing on the data in the file or the data set.
Optionally, the data processing mode is parallel processing or serial processing, and the processing module is configured to:
in response to the data processing mode being parallel processing, utilizing a parallel computing technology to process a plurality of sub-transactions in parallel to obtain a plurality of processing results;
and responding to the data processing mode as serial processing, and sequentially processing each sub-transaction according to the splitting sequence among the sub-transactions to obtain a plurality of processing results.
Optionally, the processing module is further configured to:
monitoring the processing result of each sub-transaction to call an exception handling mechanism to reprocess the target sub-transaction when the processing result of the monitored target sub-transaction is failure;
and in response to the fact that the processing result of the target sub-transaction is still failure under the condition that the preset retry times are reached, determining that the processing result is other sub-transactions which are successful, rolling back the data processed by the other sub-transactions aiming at the target database, and returning a processing failure message to the client.
To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a transaction electronic device.
The electronic equipment of the embodiment of the invention comprises: one or more processors; and a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement any of the transaction methods described above.
To achieve the above object, according to still another aspect of the embodiments of the present invention, there is provided a computer-readable medium having stored thereon a computer program which, when executed by a processor, implements any of the above-described transaction methods.
To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a computer program product. A computer program product according to an embodiment of the present invention includes a computer program that, when executed by a processor, implements a transaction processing method according to an embodiment of the present invention.
According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: the large transaction early warning mechanism is arranged, the transactions can be monitored in the running batch processing process, the early warning is carried out when the transaction exceeds a threshold value, and the transaction splitting is automatically carried out by using the splitting strategy, so that the processing efficiency and accuracy of the running batch transactions are improved, and the burden brought to a database is reduced.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a transaction processing method according to an embodiment of the present invention;
FIG. 2 is a flow diagram of an alternative transaction method according to an embodiment of the invention;
FIG. 3 is a flow diagram of an alternative transaction method according to an embodiment of the invention;
FIG. 4 is a flow diagram of yet another alternative transaction method according to an embodiment of the invention;
FIG. 5 is a schematic diagram of the main blocks of a transaction device according to an embodiment of the present invention;
FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
fig. 7 is a schematic diagram of a computer system suitable for use in implementing a mobile device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It is noted that embodiments of the invention and features of the embodiments may be combined with each other without conflict. In the technical scheme of the invention, the related aspects of acquisition, analysis, use, transmission, storage and the like of the personal information of the user accord with the regulations of related laws and regulations, are used for legal and reasonable purposes, are not shared, leaked or sold outside the legal use aspects and the like, and are subjected to supervision and management of a supervision department. Necessary measures should be taken for the personal information of the user to prevent illegal access to such personal information data, ensure that personnel having access to the personal information data comply with the regulations of the relevant laws and regulations, and ensure the personal information of the user.
Once these user personal information data are no longer needed, the risk should be minimized by limiting or even prohibiting the data collection and/or deletion. User privacy is protected, when applicable, by de-identifying the data, including in some related applications, such as by removing a particular identifier (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at a city level rather than at a specific address level), controlling how the data is stored, and/or other methods.
Referring to fig. 1, a main flowchart of a transaction processing method provided by an embodiment of the present invention is shown, including the following steps:
s101: receiving a transaction which is transmitted by a client and aims at a target database; the transaction comprises a transaction type, a data set to be processed and a data processing mode;
s102: determining a database type of a target database, reading pre-warning conditions configured for the database type in advance, and acquiring a pre-warning quantity threshold value from the pre-warning conditions;
s103: determining a transaction splitting strategy according to the transaction type to split the transaction into a plurality of sub-transactions in response to the data volume of the data set being greater than or equal to the early warning volume threshold;
s104: and processing each sub-transaction by using the data processing mode to obtain a plurality of processing results, and returning a processing success message to the client in response to the success of the plurality of processing results.
In the above embodiments, the present solution relates to the fields of data processing, batch data processing, and database management. Wherein a transaction, referred to as a database transaction, refers to a sequence of database operations accessing and operating on various data items, either all performed or none performed, is an indivisible unit of work. A transaction consists of all database operations performed between the beginning of the transaction and the end of the transaction.
In order to solve the problems of execution error, database performance reduction, system breakdown, database breakdown and the like caused by overlarge transaction submitted by a database in the batch running process under the condition of a large amount of service data or sudden burst of service data of a system, the scheme carries out early warning on the submitted transaction in the batch running process, and splits the transaction exceeding a threshold value into small transactions in a configured form, thereby avoiding the problem possibly caused by overlarge transaction.
For step S101, the system Server may set one or more databases, such as a relational database (e.g. MySQL, oracle, SQL Server), a non-relational database (e.g. MongoDB, redis, cassandra), a graph database (e.g. Neo4 j), a columnar database (e.g. HBase), a key value database (e.g. dis), etc. The client may choose to operate on only one of the databases, such as inserting a new data into the MySQL database, and this is mainly described below as MySQL.
The server receives a transaction for MySQL transmitted by the client, wherein the transaction comprises a transaction type: SQL statement/file/call external interface acquired data set, data set to be processed, data processing mode: one of parallel or serial.
For steps S102-S104, the different databases have different values for large transactions, and some databases can support very large transactions, and here, the large transactions are generally distinguished by data volume, and the early warning values need to be configured according to the different databases. Different databases are of different types, and different early warning conditions are set in advance according to different database types.
The early warning condition is realized by reading the configuration of the configuration table, and the database type and the early warning quantity threshold value are configured in the configuration table aiming at different databases. For example, the MySQL configuration table has a pre-warning number threshold of 10 ten thousand, i.e. when the number of to-be-processed data in the MySQL database exceeds 10 ten thousand, pre-warning is triggered. The early warning condition can be set according to experience or actual requirements, for example, early warning judgment can be performed according to factors such as the data size, complexity, execution time and the like of the transaction. And if the early warning condition is met, outputting early warning information.
In the prior art, if the traffic rapidly grows in a short time by means of increasing the threshold value, the set threshold value cannot ensure that the data volume of the transaction is smaller than or equal to the threshold value, and the problem of overlarge and error reporting of the transaction can be caused. Aiming at the problem, the scheme sets the early warning condition, judges the input transaction data according to the early warning quantity threshold value in the early warning condition, and outputs early warning information if the early warning condition is met.
And responding to triggering early warning, splitting the large transaction data to form a plurality of small transactions. The split policy corresponds to the transaction type, see in particular the description of figures 2 and 3 that follows. The method can split the transaction data according to the modes of time, quantity, ID range and the like, and the split transaction data can be processed in parallel or processed in series, and is determined according to the data processing mode in the transaction, so that the data processing efficiency is improved. Specifically: in response to the data processing mode being parallel processing, utilizing a parallel computing technology to process a plurality of sub-transactions in parallel to obtain a plurality of processing results; and responding to the data processing mode as serial processing, and sequentially processing each sub-transaction according to the splitting sequence among the sub-transactions to obtain a plurality of processing results.
The data processing mode is specifically configured by a user at the client, some services need to be processed sequentially, and some services need to be executed in parallel, so that sub-transaction processing is realized through the configuration mode. But whether serial or parallel, once the processing result of one sub-transaction is failed, the whole transaction processing result is failed, otherwise, the processing result is successful. The processing results need to be returned to the client.
According to the method provided by the embodiment, the large transaction early warning mechanism is set to monitor the received large transaction data volume to be processed in real time, early warning information is sent out under the condition that the early warning condition is met, the large transaction is automatically decomposed into smaller and easier to process small transactions, each transaction is guaranteed to not exceed the early warning quantity threshold value of the database, and therefore excessive pressure on system performance is avoided. The whole process does not need to realize code splitting again, solves the problem that the existing system performance is too complex, and reduces the difficulty of management and maintenance.
Referring to fig. 2, an alternative transaction method flow diagram is shown, comprising the steps of:
s201: receiving SQL sentences aiming at a target database transmitted by a client;
s202: using an SQL analysis library to analyze the SQL sentence in grammar and lexical analysis to analyze the SQL sentence into a grammar tree, and obtaining a data set to be processed in the SQL sentence and a data processing mode from the grammar tree;
s203: determining a database type of a target database, reading pre-warning conditions configured for the database type in advance, and acquiring a pre-warning quantity threshold value from the pre-warning conditions;
s204: reading metadata names from the transactions in response to the data volume of the data set being greater than or equal to the early warning number threshold, determining a metadata table corresponding to the metadata names to determine a primary key of the metadata table, and obtaining a maximum value and a minimum value of the primary key;
s205: splitting a section formed by the maximum value and the minimum value by using a preset section to obtain a plurality of sub-sections, and generating sub-transactions based on data corresponding to the sub-sections in a metadata table;
s206: and processing each sub-transaction by using the data processing mode to obtain a plurality of processing results, and returning a processing success message to the client in response to the success of the plurality of processing results.
In the above embodiment, for step S203, reference is made to the description shown in fig. 1, and no further description is given here.
For steps S201 and S202, the present embodiment also relates to the field of sql syntax parsing processing. For the case that the transaction type is SQL, that is, the client transmits an SQL statement for MySQL to the server. The method comprises the steps of identifying the content of a data set in an SQL sentence through grammar analysis, specifically, using an SQL analysis library to analyze the SQL sentence in a grammar way, analyzing the SQL sentence into a grammar tree, and obtaining the data set to be processed in the SQL sentence and a data processing mode from the grammar tree.
For steps S204-S206, assume that an updated or deleted SQL statement is DELETE FROM B, where B is the metadata name, and determine the corresponding metadata table. Assuming that the c-field is the primary key of the B-table, the data in the B-table may be ordered using the c-field to determine the maximum and minimum values of the c-field. Assume that the value of the c field is 1-100, i.e., the minimum value is 1 and the maximum value is 100. If the number of preset intervals is 10, the splitting result is 1-10, the minimum value is 1, and the maximum value is 10;11-20 is also a section, the minimum value is 11, and the maximum value is 20; … … 91-100 is also an interval.
And finally splitting 1-100 into 10 intervals, and correspondingly respectively constructing a paging metadata table through the intervals to obtain a plurality of sub-transactions. Finally, the SQL statement is split into a plurality of DELETE From B where b.c= and b.c < =: b.c > =1andbc < =10, … …, b.c > =90andbc < =100.
When splitting transactions, the boundary and the sequence of each sub-transaction need to be clear, and if the transaction is split incorrectly, the problem that the transaction boundary is unclear or the transaction sequence is incorrect may be caused. According to the scheme, the boundary unclear problem can be solved by dividing the interval and dividing the interval according to the sequence from the minimum value to the maximum value of the primary key.
According to the method provided by the embodiment, for SQL statement type transactions, no matter updating, deleting or inserting, the same processing mode is used for splitting the transactions under the condition that the transaction data volume reaches the early warning, and the code is not required to be rewritten, so that the transaction splitting boundary can be ensured, and meanwhile, the problem of complexity of splitting the transactions by re-implementing the code is solved.
Referring to FIG. 3, another alternative transaction method flow diagram is shown, comprising the steps of:
S301: receiving data to be inserted aiming at a target database and a data processing mode transmitted by a client; the data is a file uploaded by the client, or an external interface is called to acquire a data set from a data acquisition path according to the data acquisition path transmitted by the client;
s302: storing the data in the data set or the file into the temporary file row by row according to the data acquisition time sequence, and dividing the fields of each row by using a specific separator;
s303: determining a database type of a target database, reading pre-warning conditions configured for the database type in advance, and acquiring a pre-warning quantity threshold value from the pre-warning conditions;
s304: responding to the data volume of the data set being greater than or equal to the early warning number threshold, acquiring the number of lines in the temporary file, and splitting the number of lines in the temporary file by using the preset number of lines to obtain a plurality of sub-transactions;
s305: and processing each sub-transaction by using the data processing mode to obtain a plurality of processing results, and returning a processing success message to the client in response to the success of the plurality of processing results.
In the above embodiment, the steps S303 and S305 are described with reference to fig. 1, and are not described herein.
For steps S301 and S302, the data inserted for the target database stripe transmitted by the client may also be from a file, data calling an external interface, etc., such as a data synchronization interface, an access network interface. Thus, the client may upload a file, or a data acquisition path, and then invoke an external interface to acquire a data set from the data acquisition path.
And carrying out preliminary processing on the data transmitted by the client, including data cleaning, format conversion and the like, so that the data meets the format and requirements of subsequent processing. Data cleansing is mainly aimed at processing date formats, converting the date formats into standard date formats and converting unified codes into UTF-8. After the processed format of the data to be processed is finished, the data enters a temporary file to be stored in rows according to the data acquisition time sequence, and each field of a single row is divided according to a specific separator, such as "-", "/", and the like. In practice, no separator may be used.
For step S304, the temporary file is used for storing the data after format conversion, mainly for the case of inserting large amount of data, so as to prevent the memory overflow problem caused by the excessive data volume. For the temporary file, the data writing thread is required to control the size of each transaction to be divided into a plurality of submissions, and finally, the plurality of threads are finished after the submissions are finished. For data in the temporary file, data splitting can be used, for example, data processing is performed by reaching a certain line number, then batch processing parallel submission is performed by utilizing multithreading, for example, splitting into 10 sub-transactions, the number of data writing threads is 10, and one thread submits one transaction to a database for processing.
According to the method provided by the embodiment, the temporary file is split aiming at the inserted file or data stored in the temporary file, so that a split transaction mode is identified by the technology, the code is not required to be rewritten, and the problem of complexity increase caused by the existing rewritten code is solved.
Referring to fig. 4, a main flow chart of yet another alternative transaction processing method provided by an embodiment of the present invention is shown, including the following steps:
s401: receiving a transaction which is transmitted by a client and aims at a target database; the transaction comprises a transaction type, a data set to be processed and a data processing mode;
s402: determining a database type of a target database, reading pre-warning conditions configured for the database type in advance, and acquiring a pre-warning quantity threshold value from the pre-warning conditions;
s403: determining a transaction splitting strategy according to the transaction type to split the transaction into a plurality of sub-transactions in response to the data volume of the data set being greater than or equal to the early warning volume threshold;
s404: processing each sub-transaction by using the data processing mode to obtain a plurality of processing results;
s405: monitoring the processing result of each sub-transaction to call an exception handling mechanism to reprocess the target sub-transaction when the processing result of the monitored target sub-transaction is failure;
S406: in response to the fact that the processing result of the target sub-transaction is still failed under the condition that the preset retry times are reached, determining that the processing result is other sub-transactions which are successful, rolling back data processed by the other sub-transactions aiming at the target database, and returning a processing failure message to the client;
s407: and responding to the plurality of processing results to be successful, and returning a processing success message to the client.
In the above embodiment, the steps S401 to S404, S407 may be described with reference to fig. 1, and will not be described again here.
For steps S405 and S406, the present solution further monitors the processing situation of each sub-transaction, and presumes that there are 10 sub-transactions, the data processing manner is parallel processing, that is, the 10 sub-transactions need to be processed in parallel, and if it is found that the processing result of one of the sub-transactions (i.e., the target sub-transaction) is failure during the processing, a retry mechanism is triggered to automatically detect and process an abnormal problem, and re-process the target sub-transaction.
Setting preset retry times, assuming 5 times, if the target sub-transaction processing result is found to be failed after the retry is performed 5 times, the processing result of the whole transaction is indicated to be failed. Since the transaction results in failure, the transaction cannot actually act on the target database, and a rollback to the previous situation of the transaction not processing the target database is required.
For the serial data processing mode, if an exception occurs when executing the 5 th sub-transaction, a retry mechanism needs to be triggered to automatically detect and process the exception problem, and to reprocess the 5 th sub-transaction, and only if the 5 th sub-transaction is successful, the 6 th sub-transaction is executed.
Thus, if the result of the 5 th sub-transaction is found to be failed after retrying 5 times, it indicates that the result of the entire transaction is failed. Since the transaction results in failure, the transaction cannot actually act on the target database, and the data of the first 4 transactions need to be rolled back to the situation before the transaction does not process the target database.
In order to ensure the stability and reliability of the system, the method provided by the embodiment introduces an exception handling mechanism, and can automatically detect and handle problems when the execution of sub-transactions is abnormal, thereby ensuring the correctness and integrity of data processing.
Compared with the prior art, the method provided by the embodiment of the invention has at least the following beneficial effects:
1. reliability and flexibility: and a large transaction early warning mechanism is arranged to monitor the received large transaction data volume to be processed in real time, early warning information is sent out under the condition that early warning conditions are met, the large transaction is automatically decomposed into smaller and easier-to-process small transactions, and the situation that each transaction does not exceed the threshold value of the early warning number of the database is ensured, so that the system performance is prevented from being excessively stressed. The data processing process and the interaction process among the modules are optimized and designed reliably, so that the high reliability, stability and flexibility of the system are ensured.
2. The simplicity is as follows: the whole process does not need to analyze the business logic of the execution code and does not need to rewrite the business logic code, and the business can be split according to specific execution conditions by using a configuration form, so that the processing efficiency is improved and the system load is reduced. And whether the small transaction can be efficiently processed by adopting a parallel processing mechanism or not is identified, and the speed and the efficiency of data processing are greatly improved by utilizing a parallel computing technology.
3. Data consistency: in the process of transaction splitting and parallel processing, a distributed transaction framework and a consistency protocol are adopted, so that the consistency and accuracy of data are ensured. This helps ensure that each small transaction is executed correctly and eventually summarized into a complete, consistent data result.
4. Scalability: different business scenarios may require different splitting policies, and the prior art may not meet the flexible requirements of all business scenarios. The scheme adopts a distributed architecture and a modularized design, and can be conveniently expanded and upgraded. When the amount of service data increases or the processing demand increases, the device can be upgraded and expanded by modifying configuration or optimizing algorithm, etc. to meet the changing service demand.
5. Exception handling mechanisms, to ensure system stability and reliability, exception handling mechanisms are introduced. When abnormal conditions occur, problems such as data collision, loss and the like can be automatically detected and processed, so that the correctness and the integrity of data processing are ensured.
Referring to fig. 5, a schematic diagram of main modules of a transaction processing device 500 according to an embodiment of the present invention is shown, including:
a data receiving module 501, configured to receive a transaction for a target database transmitted by a client; the transaction comprises a transaction type, a data set to be processed and a data processing mode;
the early warning module 502 is configured to determine a database type of a target database, read early warning conditions configured for the database type in advance, and obtain an early warning number threshold from the early warning conditions;
a splitting module 503, configured to determine a transaction splitting policy according to the transaction type to split the transaction into a plurality of sub-transactions in response to the data volume of the data set being greater than or equal to the early warning number threshold;
and the processing module 504 is configured to process each sub-transaction by using the data processing manner, obtain a plurality of processing results, and return a processing success message to the client in response to the plurality of processing results being successful.
The data receiving module, the early warning module, the splitting module and the processing module form the main part of the device together. The data receiving module is responsible for inputting data, a program starts to enter the data receiving module to read and analyze the data, then enters the early warning module after data format conversion is carried out, the early warning module analyzes the size of a data set to be processed, judges whether a transaction early warning value (namely an early warning number threshold) set by the data is reached or not according to a preset rule, the splitting module splits a transaction according to early warning information, splits a large transaction into small transactions, and finally the processing module carries out parallel/serial submission on the split small transactions, and the paper suction pipe is ended.
The data receiving module comprises a data input interface and a data processing unit. The data input interface is for receiving data input of various sources and formats. The data processing unit judges the data source, if the data source is an SQL sentence, the data is analyzed and identified through SQL grammar, the data processing mode is adopted, if the data is derived from other data (such as data acquired by a file or a calling interface), the data is subjected to data cleaning processing firstly, then the data format is converted, the format and the requirements of subsequent processing are met, and finally the data is output to the early warning module.
The early warning module comprises an early warning condition setting unit and an early warning judging unit. The early warning condition setting unit is used for setting early warning conditions and can be set according to experience or actual requirements. The early warning judging unit judges the inputted transaction data according to the set early warning conditions, and if the early warning conditions are met, the early warning information is output.
The split module may include a split condition setting unit, a transaction split unit, and a transaction queue unit. The splitting condition setting unit is used for setting a splitting strategy. The transaction splitting unit splits the large transaction data according to the set splitting strategy to form a plurality of small transactions. The transaction queue unit puts the split small transaction data into a queue according to a certain sequence, such as a single-list field or a plurality of field value size ordering fields, and the specific fields are obtained by reading the prior configuration so as to be submitted by the processing module in parallel.
The processing module comprises a processing unit and a result output unit. The processing unit is used for carrying out parallel processing on the small transaction data in the queue and carrying out efficient processing on the small transaction data to the database by utilizing a parallel computing technology.
In the embodiment of the present invention, the transaction type is a structured query SQL statement, and the data receiving module 501 is configured to:
Receiving SQL sentences aiming at a target database and transmitted by a client, and using an SQL analysis library to analyze the SQL sentences in a grammar way so as to analyze the SQL sentences into grammar trees, and acquiring a data set to be processed and a data processing mode in the SQL sentences from the grammar trees;
the splitting module 503 is configured to:
reading a metadata name from the transaction, determining a metadata table corresponding to the metadata name, determining a primary key of the metadata table, and acquiring a maximum value and a minimum value of the primary key;
splitting a section formed by the maximum value and the minimum value by using a preset section to obtain a plurality of sub-sections, and generating sub-transactions based on data corresponding to the sub-sections in the metadata table.
In the embodiment of the present invention, the transaction type is data to be inserted, and the data receiving module 501 is configured to:
receiving data to be inserted aiming at a target database and a data processing mode transmitted by a client; the data is a file uploaded by the client, or an external interface is called to acquire a data set from a data acquisition path according to the data acquisition path transmitted by the client;
storing the data in the data set or the file into the temporary file row by row according to the data acquisition time sequence, and dividing the fields of each row by using a specific separator;
The splitting module 503 is configured to: and acquiring the number of lines in the temporary file, and splitting the number of lines in the temporary file by using the preset number of lines to obtain a plurality of sub-transactions.
In the embodiment of the present invention, the data receiving module 501 is further configured to:
and performing data cleaning and format conversion processing on the data in the file or the data set.
In the embodiment of the present invention, the data processing manner is parallel processing or serial processing, and the processing module 504 is configured to:
in response to the data processing mode being parallel processing, utilizing a parallel computing technology to process a plurality of sub-transactions in parallel to obtain a plurality of processing results;
and responding to the data processing mode as serial processing, and sequentially processing each sub-transaction according to the splitting sequence among the sub-transactions to obtain a plurality of processing results.
In the embodiment of the present invention, the processing module 504 is further configured to:
monitoring the processing result of each sub-transaction to call an exception handling mechanism to reprocess the target sub-transaction when the processing result of the monitored target sub-transaction is failure;
and in response to the fact that the processing result of the target sub-transaction is still failure under the condition that the preset retry times are reached, determining that the processing result is other sub-transactions which are successful, rolling back the data processed by the other sub-transactions aiming at the target database, and returning a processing failure message to the client.
In addition, the implementation of the apparatus in the embodiments of the present invention has been described in detail in the above method, so that the description is not repeated here.
Fig. X shows an exemplary system architecture X00, including terminal devices X01, X02, 603, a network 604 and a server 605 (by way of example only), to which embodiments of the invention may be applied.
The terminal devices 601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, are installed with various communication client applications, and a user may interact with the server 605 through the network 604 using the terminal devices 601, 602, 603 to receive or transmit messages, etc.
The network 604 is used as a medium to provide communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The server 605 may be a server providing various services, and it should be noted that, the method provided by the embodiment of the present invention is generally executed by the server 605, and accordingly, the apparatus is generally disposed in the server 605.
It should be understood that the number of terminal devices, networks and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 7, there is illustrated a schematic diagram of a computer system 700 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, ROM 702, and RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output portion 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 708 including a hard disk or the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. The drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is mounted into the storage section 708 as necessary.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 709, and/or installed from the removable medium 711. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 701.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: the processor comprises a data receiving module, an early warning module, a splitting module and a processing and fetching module. The names of these modules do not in any way constitute a limitation of the module itself, for example, a data receiving module may also be described as a "receiving module".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to perform any of the transaction methods described above.
The computer program product of the present invention comprises a computer program which, when executed by a processor, implements the transaction method of the embodiments of the present invention.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (13)
1. A method of transaction processing, comprising:
receiving a transaction which is transmitted by a client and aims at a target database; the transaction comprises a transaction type, a data set to be processed and a data processing mode;
Determining a database type of a target database, reading pre-warning conditions configured for the database type in advance, and acquiring a pre-warning quantity threshold value from the pre-warning conditions;
determining a transaction splitting strategy according to the transaction type to split the transaction into a plurality of sub-transactions in response to the data volume of the data set being greater than or equal to the early warning volume threshold;
and processing each sub-transaction by using the data processing mode to obtain a plurality of processing results, and returning a processing success message to the client in response to the success of the plurality of processing results.
2. The method of claim 1, wherein the transaction type is a structured query SQL statement, and the receiving a client-transmitted transaction against a target database comprises:
receiving SQL sentences aiming at a target database and transmitted by a client, and using an SQL analysis library to analyze the SQL sentences in a grammar way so as to analyze the SQL sentences into grammar trees, and acquiring a data set to be processed and a data processing mode in the SQL sentences from the grammar trees;
the determining a transaction splitting policy according to the transaction type to split the transaction into a plurality of sub-transactions includes:
Reading a metadata name from the transaction, determining a metadata table corresponding to the metadata name, determining a primary key of the metadata table, and acquiring a maximum value and a minimum value of the primary key;
splitting a section formed by the maximum value and the minimum value by using a preset section to obtain a plurality of sub-sections, and generating sub-transactions based on data corresponding to the sub-sections in the metadata table.
3. The method of claim 1, wherein the transaction type is data to be inserted, and the receiving a transaction transmitted by the client for the target database comprises:
receiving data to be inserted aiming at a target database and a data processing mode transmitted by a client; the data is a file uploaded by the client, or an external interface is called to acquire a data set from a data acquisition path according to the data acquisition path transmitted by the client;
storing the data in the data set or the file into the temporary file row by row according to the data acquisition time sequence, and dividing the fields of each row by using a specific separator;
determining a transaction splitting policy according to the transaction type to split the transaction into a plurality of sub-transactions, including:
and acquiring the number of lines in the temporary file, and splitting the number of lines in the temporary file by using the preset number of lines to obtain a plurality of sub-transactions.
4. A method according to claim 3, characterized in that the method further comprises:
and performing data cleaning and format conversion processing on the data in the file or the data set.
5. The method according to claim 1, wherein the data processing mode is parallel processing or serial processing, and the processing each sub-transaction using the data processing mode obtains a plurality of processing results, including:
in response to the data processing mode being parallel processing, utilizing a parallel computing technology to process a plurality of sub-transactions in parallel to obtain a plurality of processing results;
and responding to the data processing mode as serial processing, and sequentially processing each sub-transaction according to the splitting sequence among the sub-transactions to obtain a plurality of processing results.
6. The method according to claim 1, wherein the method further comprises:
monitoring the processing result of each sub-transaction to call an exception handling mechanism to reprocess the target sub-transaction when the processing result of the monitored target sub-transaction is failure;
and in response to the fact that the processing result of the target sub-transaction is still failure under the condition that the preset retry times are reached, determining that the processing result is other sub-transactions which are successful, rolling back the data processed by the other sub-transactions aiming at the target database, and returning a processing failure message to the client.
7. A transaction processing device, comprising:
the data receiving module is used for receiving the transaction which is transmitted by the client and aims at the target database; the transaction comprises a transaction type, a data set to be processed and a data processing mode;
the early warning module is used for determining the database type of the target database, reading early warning conditions configured for the database type in advance, and acquiring early warning quantity threshold values from the early warning conditions;
the splitting module is used for determining a transaction splitting strategy according to the transaction type in response to the data volume of the data set being greater than or equal to the early warning number threshold value so as to split the transaction into a plurality of sub-transactions;
and the processing module is used for processing each sub-transaction by using the data processing mode to obtain a plurality of processing results, and returning a processing success message to the client in response to the success of the plurality of processing results.
8. The apparatus of claim 7, wherein the transaction type is a structured query SQL statement, and wherein the data receiving module is configured to:
receiving SQL sentences aiming at a target database and transmitted by a client, and using an SQL analysis library to analyze the SQL sentences in a grammar way so as to analyze the SQL sentences into grammar trees, and acquiring a data set to be processed and a data processing mode in the SQL sentences from the grammar trees;
The splitting module is used for:
reading a metadata name from the transaction, determining a metadata table corresponding to the metadata name, determining a primary key of the metadata table, and acquiring a maximum value and a minimum value of the primary key;
splitting a section formed by the maximum value and the minimum value by using a preset section to obtain a plurality of sub-sections, and generating sub-transactions based on data corresponding to the sub-sections in the metadata table.
9. The apparatus of claim 7, wherein the transaction type is data to be inserted, and wherein the data receiving module is configured to:
receiving data to be inserted aiming at a target database and a data processing mode transmitted by a client; the data is a file uploaded by the client, or an external interface is called to acquire a data set from a data acquisition path according to the data acquisition path transmitted by the client;
storing the data in the data set or the file into the temporary file row by row according to the data acquisition time sequence, and dividing the fields of each row by using a specific separator;
the splitting module is used for:
and acquiring the number of lines in the temporary file, and splitting the number of lines in the temporary file by using the preset number of lines to obtain a plurality of sub-transactions.
10. The apparatus of claim 7, wherein the processing module is further configured to:
monitoring the processing result of each sub-transaction to call an exception handling mechanism to reprocess the target sub-transaction when the processing result of the monitored target sub-transaction is failure;
and in response to the fact that the processing result of the target sub-transaction is still failure under the condition that the preset retry times are reached, determining that the processing result is other sub-transactions which are successful, rolling back the data processed by the other sub-transactions aiming at the target database, and returning a processing failure message to the client.
11. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-6.
12. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-6.
13. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410147781.0A CN117667319B (en) | 2024-02-02 | 2024-02-02 | Transaction processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410147781.0A CN117667319B (en) | 2024-02-02 | 2024-02-02 | Transaction processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117667319A true CN117667319A (en) | 2024-03-08 |
| CN117667319B CN117667319B (en) | 2024-05-03 |
Family
ID=90075419
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410147781.0A Active CN117667319B (en) | 2024-02-02 | 2024-02-02 | Transaction processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117667319B (en) |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130232177A1 (en) * | 2010-09-28 | 2013-09-05 | Yiftach Shoolman | Systems, methods, and media for managing ram resources for in-memory nosql databases |
| US20140129531A1 (en) * | 2012-11-07 | 2014-05-08 | Nec Laboratories America, Inc. | Automatic Entity-Grouping for Scalable OLTP |
| US9183200B1 (en) * | 2012-08-02 | 2015-11-10 | Symantec Corporation | Scale up deduplication engine via efficient partitioning |
| CN110019468A (en) * | 2017-12-05 | 2019-07-16 | 华为技术有限公司 | A kind of Database Systems and data bank access method |
| CN110546609A (en) * | 2017-04-19 | 2019-12-06 | 华为技术有限公司 | Hardware Transactional Memory (HTM) assisted database transactions |
| CN112306646A (en) * | 2020-06-29 | 2021-02-02 | 北京沃东天骏信息技术有限公司 | Method, device, equipment and readable storage medium for processing transaction |
| CN112883045A (en) * | 2021-03-31 | 2021-06-01 | 中国工商银行股份有限公司 | Database transaction splitting execution method and device |
| WO2021109719A1 (en) * | 2019-12-03 | 2021-06-10 | 腾讯科技（深圳）有限公司 | Transaction processing method, apparatus, and device, and computer storage medium |
| CN113609152A (en) * | 2021-06-30 | 2021-11-05 | 天津五八到家货运服务有限公司 | Data processing method and device and computing equipment |
| CN113806312A (en) * | 2021-09-28 | 2021-12-17 | 北京京东拓先科技有限公司 | File processing method and device, electronic equipment and storage medium |
| CN114253675A (en) * | 2021-12-23 | 2022-03-29 | 北京人大金仓信息技术股份有限公司 | Transaction processing method, device, equipment and storage medium |
| CN115408411A (en) * | 2022-09-01 | 2022-11-29 | 阿里云计算有限公司 | Data writing method and device, electronic equipment and storage medium |
| US20230090266A1 (en) * | 2021-09-22 | 2023-03-23 | International Business Machines Corporation | Managing transaction size during database replication |
| CN116303524A (en) * | 2022-12-08 | 2023-06-23 | 中国农业银行股份有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN116541461A (en) * | 2023-04-03 | 2023-08-04 | 中国工商银行股份有限公司 | Data processing method, device, equipment and storage medium applied to database |
| US11748212B1 (en) * | 2021-06-28 | 2023-09-05 | Gravic, Inc. | Method and apparatus for resolving automatic transaction facility (ATF) failures |
-
2024
- 2024-02-02 CN CN202410147781.0A patent/CN117667319B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130232177A1 (en) * | 2010-09-28 | 2013-09-05 | Yiftach Shoolman | Systems, methods, and media for managing ram resources for in-memory nosql databases |
| US9183200B1 (en) * | 2012-08-02 | 2015-11-10 | Symantec Corporation | Scale up deduplication engine via efficient partitioning |
| US20140129531A1 (en) * | 2012-11-07 | 2014-05-08 | Nec Laboratories America, Inc. | Automatic Entity-Grouping for Scalable OLTP |
| CN110546609A (en) * | 2017-04-19 | 2019-12-06 | 华为技术有限公司 | Hardware Transactional Memory (HTM) assisted database transactions |
| CN110019468A (en) * | 2017-12-05 | 2019-07-16 | 华为技术有限公司 | A kind of Database Systems and data bank access method |
| WO2021109719A1 (en) * | 2019-12-03 | 2021-06-10 | 腾讯科技（深圳）有限公司 | Transaction processing method, apparatus, and device, and computer storage medium |
| CN112306646A (en) * | 2020-06-29 | 2021-02-02 | 北京沃东天骏信息技术有限公司 | Method, device, equipment and readable storage medium for processing transaction |
| CN112883045A (en) * | 2021-03-31 | 2021-06-01 | 中国工商银行股份有限公司 | Database transaction splitting execution method and device |
| US11748212B1 (en) * | 2021-06-28 | 2023-09-05 | Gravic, Inc. | Method and apparatus for resolving automatic transaction facility (ATF) failures |
| CN113609152A (en) * | 2021-06-30 | 2021-11-05 | 天津五八到家货运服务有限公司 | Data processing method and device and computing equipment |
| US20230090266A1 (en) * | 2021-09-22 | 2023-03-23 | International Business Machines Corporation | Managing transaction size during database replication |
| CN113806312A (en) * | 2021-09-28 | 2021-12-17 | 北京京东拓先科技有限公司 | File processing method and device, electronic equipment and storage medium |
| CN114253675A (en) * | 2021-12-23 | 2022-03-29 | 北京人大金仓信息技术股份有限公司 | Transaction processing method, device, equipment and storage medium |
| CN115408411A (en) * | 2022-09-01 | 2022-11-29 | 阿里云计算有限公司 | Data writing method and device, electronic equipment and storage medium |
| CN116303524A (en) * | 2022-12-08 | 2023-06-23 | 中国农业银行股份有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN116541461A (en) * | 2023-04-03 | 2023-08-04 | 中国工商银行股份有限公司 | Data processing method, device, equipment and storage medium applied to database |
Non-Patent Citations (3)
| Title |
|---|
| 周松华;熊勇;: "数据库查询优化策略", 井冈山学院学报, no. 07, 30 January 2006 (2006-01-30) * |
| 孙晶, 李晓明, 张莲梅, 胡启平: "电能量远程采集数据并发存取过程中Sybase数据库防死锁的设计", 计算机工程与应用, no. 15, 21 May 2003 (2003-05-21) * |
| 爱可生开源社区: "技术分享 | TiDB 对大事务的简单拆分", Retrieved from the Internet <URL:https://blog.csdn.net/ActionTech/article/details/120042180> * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117667319B (en) | 2024-05-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| US20210049087A1 (en) | Data monitoring methods, apparatuses, electronic devices, and computer readable storage media | |
| CN111338834B (en) | Data storage method and device | |
| CN113051319A (en) | Redis-based large key detection method, system, device and storage medium | |
| CN109325002B (en) | Text file processing method, device and system, electronic equipment and storage medium | |
| CN117667319B (en) | Transaction processing method and device | |
| CN112579632A (en) | Data verification method, device, equipment and medium | |
| CN116204428A (en) | Test case generation method and device | |
| CN112148762A (en) | Statistical method and device for real-time data stream | |
| CN116028243A (en) | Message processing method and device | |
| CN111224756B (en) | Method and device for determining data transmission abnormity, storage medium and electronic equipment | |
| CN114490050A (en) | Data synchronization method and device | |
| CN114328464A (en) | Data maintenance method, device, equipment and readable medium for distributed storage equipment | |
| CN113656239A (en) | Monitoring method and device for middleware and computer program product | |
| CN108628909B (en) | Information pushing method and device | |
| CN113742355A (en) | Method, device, equipment and computer readable medium for updating inventory | |
| CN112732728A (en) | Data synchronization method and system | |
| CN113779048A (en) | Data processing method and device | |
| CN112000720A (en) | Management method and management system for database connection and database connection pool | |
| CN111290927A (en) | Data monitoring method and device | |
| CN117331920A (en) | Data quality detection processing method and device | |
| CN112579717B (en) | Method and device for displaying user address on electronic map | |
| CN112398885B (en) | Data transmission method and device | |
| WO2021047506A1 (en) | System and method for statistical analysis of data, and computer-readable storage medium | |
| CN113807054A (en) | Method, device, equipment and computer readable medium for processing data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |