CN116089441A

CN116089441A - Batch data updating method, batch data updating device and batch data updating system

Info

Publication number: CN116089441A
Application number: CN202211733628.3A
Authority: CN
Inventors: 汤雄; 何云; 马银剑; 何文瀚; 周辉
Original assignee: Postal Savings Bank of China Ltd
Current assignee: Postal Savings Bank of China Ltd
Priority date: 2022-12-30
Filing date: 2022-12-30
Publication date: 2023-05-09

Abstract

The application provides a batch data updating method, a batch data updating device and a batch data updating system. The updating method comprises the following steps: a blocking step, namely blocking a plurality of pieces of preset service data to obtain a plurality of service data blocks; initializing, namely starting a transaction and initializing a JVM local cache; processing a plurality of pieces of preset business data in a corresponding business data block based on business logic to obtain a target business block, storing the multi-item target business data in the target business block in a JVM local cache, and updating the multi-item target business data in the JVM local cache to a target database in batches and closing the transaction; and repeating the steps, namely sequentially repeating the initializing step and the processing step at least once until all the business data blocks are processed, thereby solving the problem of higher cost for updating the batch data in the prior art.

Description

Batch data updating method, batch data updating device and batch data updating system

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a batch data updating method, an updating device, a computer readable storage medium, and a batch data updating system.

Background

At present, batch updating of databases is generally carried out by adopting a Redis database as a cache and establishing mapping with MySQL and other basic databases. By reading and writing the Redis database, the data in the Redis database is updated to the MySQL database at regular time, so that batch updating of the MySQL database is realized, the access pressure of the MySQL database can be reduced, and the transaction efficiency is further improved.

However, redis plays an important role in guaranteeing cache consistency among different nodes as a solution for distributed cache consistency. The parameter data shared by different nodes can be stored in the Redis database, so that the transaction efficiency can be improved, and the consistency of access of the cache data by different nodes can be ensured. However, for the update of single-node data in a distributed, database-division and table-division scene, the operations of updating, inserting, deleting and the like of the data cannot influence other nodes because the part of data is exclusively shared by the single nodes. If the Redis cluster is used as a cache database, the advantage of cache consistency of the Redis cluster is difficult to be exerted, the overall deployment cost of the Redis database is high, and a certain network bandwidth is required to be consumed.

Disclosure of Invention

The main objective of the present application is to provide a batch data updating method, an updating device, a computer readable storage medium and a batch data updating system, so as to solve the problem of high cost of batch data updating in the prior art.

According to an aspect of the embodiment of the present invention, there is provided a method for updating batch data, including: a blocking step, namely blocking a plurality of pieces of preset service data to obtain a plurality of service data blocks, wherein one service data block comprises a plurality of pieces of preset service data; initializing, namely starting a transaction and initializing a JVM local cache; processing, based on service logic, a plurality of pieces of preset service data in a corresponding service data block to obtain a target service block, storing the multi-item target service data in the target service block in the JVM local cache, and updating the plurality of pieces of target service data in the JVM local cache to a target database in batches and closing the transaction, wherein the target service data is service data obtained by processing the preset service data based on the service logic; and repeating the initializing step and the processing step at least once in sequence until all the service data blocks are processed.

Optionally, based on service logic, processing a plurality of pieces of predetermined service data in a corresponding one of the service data blocks to obtain a target service block, and storing the multi-item target service data in the target service block in the JVM local cache, including: based on the service logic, processing a plurality of pieces of preset service data in a corresponding service data block piece by piece to obtain a plurality of pieces of target service data; intercepting a plurality of pieces of target business data so that the plurality of pieces of target business data are not updated into the target database; and forming the target service block by a plurality of pieces of target service data, and storing the plurality of pieces of target service data in the target service block in the JVM local cache.

Optionally, intercepting the plurality of target service data so that the plurality of target service data is not updated into the target database includes: intercepting MyBatis updating operation, myBatis inserting operation and MyBatis deleting operation of the target business data so that the target business data are not updated in the target database.

Optionally, batch updating the plurality of target business data in the JVM local cache into a target database includes: and at least calling Java database connection to update a plurality of target business data in the JVM local cache to the target database in batches.

Optionally, at least invoking a Java database connection to update a plurality of the target service data in the JVM local cache into the target database in batch, including: and calling a batch updating method, a batch inserting method and a batch deleting method of the Java database connection to update a plurality of target business data in the JVM local cache to the target database in batches.

Optionally, the partitioning of the plurality of predetermined service data to obtain a plurality of service data blocks includes: under the condition of receiving a service data request, screening service data meeting the service data request from the target database to obtain a plurality of pieces of preset service data; and according to a preset rule, partitioning the plurality of pieces of preset business data to obtain a plurality of business data blocks, wherein the preset rule is a rule for partitioning the plurality of pieces of preset business data.

Optionally, after batch updating the plurality of target business data in the JVM local cache into a target database, before closing the transaction, the updating method further comprises: and clearing a plurality of pieces of target business data in the JVM local cache.

According to another aspect of the embodiment of the present invention, there is also provided an apparatus for updating batch data, including: the block dividing unit is used for executing a block dividing step and dividing a plurality of pieces of preset service data to obtain a plurality of service data blocks, wherein one service data block comprises a plurality of pieces of preset service data; the initialization unit is used for executing an initialization step, starting a transaction and initializing a JVM local cache; the processing unit is used for executing the processing steps, processing a plurality of pieces of preset business data in a corresponding business data block based on business logic to obtain a target business block, storing a plurality of pieces of target business data in the target business block in the JVM local cache, updating a plurality of pieces of target business data in the JVM local cache to a target database in batches, and closing the transaction, wherein the target business data is business data after processing the preset business data based on the business logic; and the repeating unit is used for executing the repeating steps, and sequentially repeating the initializing step and the processing step at least once until the processing of all the service data blocks is completed.

According to still another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium including a stored program, wherein the program performs any one of the batch data updating methods.

According to still another aspect of the embodiment of the present invention, there is also provided a batch data updating system, including: the UI is used for sending the business data request to the batch data updating device, and the batch data updating device is in communication connection with the UI and is used for executing any batch data updating method; and the target database is communicated with the batch data updating device.

In the batch data updating method, firstly, a plurality of pieces of preset business data are segmented to obtain a plurality of business data blocks; then, starting a transaction, and initializing a JVM local cache; then, based on service logic, processing a plurality of pieces of preset service data in a corresponding service data block to obtain a target service block, storing the multi-item target service data in the target service block in a JVM local cache, and updating the multi-item target service data in the JVM local cache to a target database in batches; finally, the initializing step and the processing step are sequentially executed at least once until the processing of all the service data blocks is completed. Compared with the prior art that Redis cache is adopted to update the multi-item label business data after business logic processing in batches, the method and the device are based on JVM local cache and transactions, namely the JVM local cache is limited in the transactions, and batch update is carried out on the multi-item label business data in the JVM local cache, so that the problem of data loss in the JVM local cache caused by difficulty in persistence of the JVM local cache can be avoided. Meanwhile, the JVM local cache is lighter and efficient, complex deployment and high deployment cost are not required, and network bandwidth occupation is not required, so that the method for updating the batch data is simpler and lower in cost, and the problem of higher cost for updating the batch data in the prior art is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:

FIG. 1 illustrates a flow chart of a method of updating batch data according to one embodiment of the present application;

FIG. 2 illustrates a flow chart of a method of updating batch data according to yet another embodiment of the present application;

FIG. 3 is a schematic structural diagram of a batch data updating device according to an embodiment of the present application;

FIG. 4 illustrates a flow chart of an update scheme for batch data in accordance with one embodiment of the present application.

Detailed Description

It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

In order to make the present application solution better understood by those skilled in the art, the following description will be made in detail and with reference to the accompanying drawings in the embodiments of the present application, it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, shall fall within the scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate in order to describe the embodiments of the present application described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

For convenience of description, the following will describe some terms or terms related to the embodiments of the present application:

transaction: a series of operations of the operation database, which are either all executed or not all not executed, are an indivisible working unit;

JVM local cache: the JVM (Java virtual machine, java Virtual Machine, JVM for short) has built-in cache, and memory for storing data throughout the program life cycle.

As described in the background art, in order to solve the foregoing problem, in an exemplary embodiment of the present application, a method for updating batch data, an updating device, a computer readable storage medium and a system for updating batch data are provided.

According to an embodiment of the application, a batch data updating method is provided.

FIG. 1 is a flow chart of a method of updating batch data according to an embodiment of the present application. As shown in fig. 1, the updating method includes the steps of:

step S101, a blocking step, namely, blocking a plurality of pieces of preset service data to obtain a plurality of service data blocks, wherein one service data block comprises a plurality of pieces of preset service data;

step S102, an initialization step, namely starting a transaction and initializing a JVM local cache;

step S103, processing step, based on business logic, processing a plurality of pieces of preset business data in a corresponding business data block to obtain a target business block, storing a plurality of pieces of target business data in the target business block in the JVM local cache, updating a plurality of pieces of target business data in the JVM local cache to a target database in batches, closing the transaction, wherein the target business data is business data processed by the preset business data based on the business logic;

Step S104, repeating the steps, and repeating the initializing step and the processing step at least once in turn until the processing of all the service data blocks is completed.

In the batch data updating method, firstly, a plurality of pieces of preset business data are segmented to obtain a plurality of business data blocks; then, starting a transaction, and initializing a JVM local cache; then, based on service logic, processing a plurality of pieces of preset service data in a corresponding service data block to obtain a target service block, storing a plurality of pieces of target service data in the target service block in a JVM local cache, and updating a plurality of pieces of target service data in the JVM local cache to a target database in batches; finally, the initializing step and the processing step are sequentially executed at least once until the processing of all the service data blocks is completed. Compared with the prior art that Redis cache is adopted to update the multi-item label business data after business logic processing in batches, the method and the device are based on JVM local cache and transactions, namely the JVM local cache is limited in the transactions, and batch update is carried out on the multi-item label business data in the JVM local cache, so that the problem of data loss in the JVM local cache caused by difficulty in persistence of the JVM local cache can be avoided. Meanwhile, the JVM local cache is lighter and efficient, complex deployment and high deployment cost are not required, and network bandwidth occupation is not required, so that the method for updating the batch data is simpler and lower in cost, and the problem of higher cost for updating the batch data in the prior art is solved.

Specifically, in the above embodiment, a plurality of pieces of predetermined service data are segmented to obtain a plurality of service data blocks. And limiting each business data block into a transaction, namely after processing a plurality of pieces of preset business data in one business data block, storing the processed corresponding target business data in a JVM local cache until all pieces of preset business data in one business data block are processed, and obtaining the target data block. And then, the multi-item target business data in one target data block are updated to the target database in batches. If the preset business data processing in the business data block fails, the preset business data in the business data block fails to roll back. When the next task is started, a plurality of pieces of preset business data in the business data block which fails to be processed last time can be pulled up again, and the processing is performed again. Meanwhile, the batch updating method can also support abnormal skipping. For example, failure of a predetermined one of the service data blocks may result in failure of the entire service data block and the entire thread. In order to avoid the occurrence of the above situation, the batch updating method of the application can also support the skip of the appointed exception, and ensure the success rate of data processing.

Specifically, in the above embodiment, the processing is performed on the plurality of pieces of predetermined service data in the corresponding service data block based on the service logic, that is, the processing is performed on the plurality of pieces of predetermined service data in the corresponding service data block based on the service logic. I.e. a transaction may correspond to a service data block. However, in the actual application process, if the service logic corresponding to the two service data blocks has an association relationship, and multiple predetermined service data in the service data blocks need to be processed at the same time, one transaction may correspond to two or two service data blocks. That is, the number of service data blocks corresponding to the above-mentioned one transaction is not limited in the present application.

Specifically, the batch data updating method can also support concurrent processing, i.e. a plurality of threads can be started for batch processing. For example, each thread may process predetermined traffic data for one or more traffic data blocks. In the actual application process, in the process of updating batch data in a concurrent mode, the number of enabled threads can be set according to the actual requirements of the service. Meanwhile, the number of the predetermined service data in each service data block can also be set according to the actual requirements of the service. For example. And if hundreds of thousands of pieces of preset business data are required to be updated in batches, 10 threads can be started for concurrent processing. Wherein each thread can process 1 ten thousand pieces of data, the predetermined service data in each service data block can be set to 200, i.e., 50 service data blocks are processed per thread. But is not limited to, the above-listed cases, the specific number of service data blocks that each thread can process, and the size of the predetermined service data in each service data block are configurable.

Specifically, any suitable method in the prior art may be used to update the multi-label business data in the JVM local cache to the target database in batches. In this application, the specific method for batch updating the multi-entry target service data in the JVM local cache to the target database is not limited.

In a specific embodiment of the present application, the target database is a database storing a plurality of pieces of predetermined service data. The target database may be a MySQL database. Of course, the target database is not limited to MySQL database, but may be any other type of suitable database in the prior art.

In an actual application process, as shown in fig. 2, in the case of receiving a request for reading data (i.e., a service data request mentioned later) sent by a user through a UI interface, a plurality of pieces of predetermined service data (i.e., read data) may be read from a target database. And the read pieces of predetermined service data are blocked (in the case that the read amount of predetermined service data is large, the read pieces of predetermined service data can be blocked, and in the case that the read amount of predetermined service data is small, the read pieces of predetermined service data can not be blocked, so that the step is not necessary), and a plurality of service data blocks are obtained. At this point, a transaction needs to be started and the JVM local cache is initialized. And then, processing (namely, processing data) a plurality of preset business data in the corresponding one or more business data blocks based on the business logic to obtain a corresponding target business block. Meanwhile, in the process of processing a plurality of pieces of preset service data, myBatis write-back operation interception (namely MyBatis update operation, myBatis insert operation and MyBatis delete operation) needs to be synchronously performed, and the multi-item target service data in the target service block is stored in a JVM local cache. And finally, writing data, namely updating the multi-item mark business data in the JVM local cache to a target database in batches, and closing the transaction.

Specifically, in the above embodiment, there may be only one predetermined service data in one service data block. That is, in the present application, the data amount of the predetermined service data in the service data block is not limited. Specifically, the service logic may be logic for processing the predetermined service data. For example, in a banking scenario, deposit data of a batch of customers needs to be closed (settlement interest). The business logic may proceed with the business logic of the interest.

It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.

In one embodiment of the present application, based on service logic, processing a plurality of pieces of the predetermined service data in a corresponding one of the service data blocks to obtain a target service block, and storing the plurality of pieces of the target service data in the target service block in the JVM local cache, including: based on the service logic, processing a plurality of pieces of preset service data in a corresponding service data block piece by piece to obtain a plurality of pieces of target service data; intercepting a plurality of pieces of target business data so that the plurality of pieces of target business data are not updated in the target database; and forming the target service block by a plurality of target service data, and storing the target service data in the target service block in the JVM local cache. In the actual application process, in the process of carrying out piece-by-piece processing on a plurality of pieces of preset business data in corresponding business data blocks based on business logic, the obtained corresponding target business data can be updated into a target database at any time, that is, the target database is continuously connected and accessed, so that the connection and access pressures of the target database are higher. Therefore, in order to reduce the connection and access pressure of the target database, the target business data processed based on the business logic is intercepted, so that the target business data processed based on the business logic cannot be updated into the target database, and the connection and access pressure of the target database can be reduced.

Specifically, in order to further more simply avoid updating the target service data into the target database, in another embodiment of the present application, intercepting a plurality of pieces of the target service data so that the plurality of pieces of the target service data are not updated into the target database includes: intercepting MyBatis updating operation, myBatis inserting operation and MyBatis deleting operation of the target business data so that the target business data are not updated in the target database.

In the actual application process, the multi-item target service data is not limited to be updated to the target database according to the corresponding target service block through MyBatis write-back operation (namely MyBatis update operation, myBatis insert operation and MyBatis delete operation). The corresponding target service block can also be updated to the target database by adopting any feasible write-back mode in the prior art. Meanwhile, under the condition that the corresponding write-back operation is changed, the corresponding interception operation also needs to be synchronously changed. The purpose of continuously connecting and accessing the target database is achieved only by modifying a plurality of pieces of preset business data. That is, the target database can be uniformly connected once after all the predetermined service data in the same service data block are processed, so that the multi-item target service data in one target service block can be updated to the target database in batches.

In another embodiment of the present application, batch updating the target service data in the JVM local cache to a target database includes: at least a Java database connection (Java Database Connectivity, JDBC for short) is invoked to batch update the plurality of target business data in the JVM local cache into the target database. In the embodiment, the target business data in the local buffer memory of the JVM is updated to the target database in batches through Java database connection, so that a plurality of business data in one target business block are updated to the target database in batches more simply and conveniently.

Of course, in the actual application process, the target service data in the JVM local cache is not limited to be updated to the target database in batches through Java database connection. The batch updating of the target business data in the JVM local cache to the target database can also be achieved by any feasible way of connecting the target database in the prior art.

In order to further more simply and conveniently update the multi-item target service data in one target service block to a target database in batches, that is, further more simply and conveniently implement batch writing back of the multi-item target service data to the target database, in yet another embodiment of the present application, at least invoking a Java database connection to update a plurality of the target service data in the JVM local cache to the target database in batches includes: and calling a batch updating method, a batch inserting method and a batch deleting method of the Java database connection to update the target business data in the JVM local cache to the target database in batches.

Specifically, in the actual application process, when the method of connecting the target database is changed, the specific method of writing (batch update method, batch insert method, and batch delete method) into the target database is also changed correspondingly.

In one embodiment of the present application, partitioning a plurality of predetermined service data to obtain a plurality of service data blocks includes: under the condition of receiving a service data request, screening service data meeting the service data request from the target database to obtain a plurality of pieces of preset service data; and dividing the predetermined business data into blocks according to a predetermined rule, wherein the predetermined rule is a rule for dividing the predetermined business data into blocks. In this embodiment, the plurality of pieces of predetermined service data are segmented according to the predetermined rule, so that the logic for partitioning the plurality of pieces of predetermined service data is simpler, the calculation amount is reduced, and the batch data updating method is lighter.

Specifically, the predetermined rule may be divided according to the number of pieces of predetermined service data. Of course, the predetermined rule may be other rules for blocking, such as date, amount, and the like. That is, the specific content of the above predetermined rule is not limited in the present application, and may be flexibly adjusted according to actual situations.

In a specific embodiment of the present application, when a total of 1000 pieces of predetermined service data are read, 200 pieces of predetermined service data may be taken as a unit, so that the total of 1000 pieces of predetermined service data may be divided into 5 blocks, i.e. 5 service data blocks are obtained.

Specifically, in the above embodiment, in the case of receiving a service data request, the process of screening data that satisfies the service data request from the target database may further include: and screening the preset service data meeting the service data request according to the service field and the idempotent field. Specifically, the service field may be a field related to a service data request, and the idempotent field may be a field for marking a state of predetermined service data or a service data block. In a specific embodiment of the present application, the predetermined traffic data is processed based on the corresponding traffic logic, and the idempotent fields (e.g., modified to processed) of the corresponding predetermined traffic data may be modified. Of course, in the practical application process, the idempotent fields can be processed more flexibly. Specifically, the design can be performed according to specific business requirements. For the preset business data which only needs to initiate batch data processing task once, the business data can be designed to be unprocessed and processed; for preset business data (such as settlement of a banking system once every three months) needing to initiate batch data processing at fixed time, the date of the settlement can be designed to be the next time; when the current date is equal to the next settlement date, a batch task is initiated, the next settlement date is modified to be a date after three months from the current date after the successful processing, the next batch task is waited again, and the like. That is, in the present application, the specific design manner of the idempotent field is not limited, and it may be flexibly set according to the characteristics of the corresponding service requirement, and so on.

In the actual application process, if the processing of the preset business data in the business data block fails, rollback is carried out. In this case, the cause of the processing failure may also be presented on the front-end UI page. In the case of repairing the cause of the processing failure (i.e., ensuring that the predetermined service data of the processing failure is successfully processed next time), the next batch processing task can be initiated. And under the condition that the next batch task is initiated, still combining the idempotent fields and the business fields for screening. For example, the idempotent field is filtered out as unprocessed scheduled service data until all scheduled service data are successfully processed.

In yet another embodiment of the present application, after batch updating the target service data in the JVM local cache into a target database, before closing the transaction, the updating method further includes: and clearing a plurality of target service data in the JVM local cache, so that the occupied JVM local cache can be released in time, and the high robustness of the whole system is ensured.

The embodiment of the application also provides a batch data updating device, and the batch data updating device can be used for executing the batch data updating method. The following describes a batch data updating device provided in the embodiment of the present application.

Fig. 3 is a schematic structural diagram of a batch data updating device according to an embodiment of the present application. As shown in fig. 3, the updating apparatus includes:

a blocking unit 10, configured to perform a blocking step, and block a plurality of predetermined service data to obtain a plurality of service data blocks, where one service data block includes a plurality of predetermined service data;

an initializing unit 20, configured to perform an initializing step, start a transaction, and initialize a JVM local cache;

a processing unit 30, configured to perform a processing step, based on a service logic, of processing a plurality of pieces of the predetermined service data in a corresponding one of the service data blocks to obtain a target service block, store a plurality of pieces of the target service data in the target service block in the JVM local cache, update a plurality of pieces of the target service data in the JVM local cache in a target database in batch, and close the transaction, where the target service data is service data obtained by processing the predetermined service data based on the service logic;

and a repeating unit 40 for executing a repeating step of sequentially repeating the initializing step and the processing step at least once until the processing of all the service data blocks is completed.

In the batch data updating device, the blocking unit is used for blocking a plurality of pieces of preset service data to obtain a plurality of service data blocks; the initialization unit is used for starting a transaction and initializing the JVM local cache; the processing unit is used for processing a plurality of pieces of preset business data in corresponding business data blocks based on business logic to obtain target business blocks, storing the multi-item target business data in the target business blocks in the JVM local cache, updating a plurality of pieces of target business data in the JVM local cache to the target database in batches, and the repeating unit is used for sequentially executing the initialization step and the processing step at least once until all business data blocks are processed. Compared with the prior art that Redis cache is adopted to update the multi-item label business data after business logic processing in batches, the method and the device are based on JVM local cache and transactions, namely the JVM local cache is limited in the transactions, and batch update is carried out on the multi-item label business data in the JVM local cache, so that the problem of data loss in the JVM local cache caused by difficulty in persistence of the JVM local cache can be avoided. Meanwhile, the JVM local cache is lighter and efficient, complex deployment and high deployment cost are not required, and network bandwidth occupation is not required, so that the method for updating the batch data is simpler and lower in cost, and the problem of higher cost for updating the batch data in the prior art is solved.

Specifically, in the above embodiment, a plurality of pieces of predetermined service data are segmented to obtain a plurality of service data blocks. And limiting each business data block into a transaction, namely after processing a plurality of pieces of preset business data in one business data block, storing the processed corresponding target business data in a JVM local cache until all pieces of preset business data in one business data block are processed, and obtaining the target data block. And then, the multi-item target business data in one target data block are updated to the target database in batches. If the preset business data processing in the business data block fails, the preset business data in the business data block fails to roll back. When the next task is started, a plurality of pieces of preset business data in the business data block which fails to be processed last time can be pulled up again, and the processing is performed again. Meanwhile, the batch updating device can also support abnormal skipping. For example, failure of a predetermined one of the service data blocks may result in failure of the entire service data block and the entire thread. In order to avoid the occurrence of the above situation, the batch updating device of the application can also support the skip of the appointed exception, and ensure the success rate of data processing.

Specifically, in the above embodiment, the processing is performed on the plurality of pieces of predetermined service data in the corresponding service data block based on the service logic, that is, the processing is performed on the plurality of pieces of predetermined service data in the corresponding service data block based on the service logic. I.e. a transaction may correspond to a block of traffic data. However, in the actual application process, if the service logic corresponding to the two service data blocks has an association relationship, and multiple predetermined service data in the service data blocks need to be processed at the same time, one transaction may correspond to two or two service data blocks. That is, the number of service data blocks corresponding to the above-mentioned one transaction is not limited in the present application.

Specifically, the batch data updating device can also support concurrent processing, i.e. a plurality of threads can be started for batch processing. For example, each thread may process predetermined traffic data for one or more traffic data blocks. In the actual application process, in the process of updating batch data in a concurrent mode, the number of enabled threads can be set according to the actual requirements of the service. Meanwhile, the number of the predetermined service data in each service data block can also be set according to the actual requirements of the service. For example. And if hundreds of thousands of pieces of preset business data are required to be updated in batches, 10 threads can be started for concurrent processing. Wherein each thread can process 1 ten thousand pieces of data, the predetermined service data in each service data block can be set to 200, i.e., 50 service data blocks are processed per thread. But is not limited to, the above-listed cases, the specific number of service data blocks that each thread can process, and the size of the predetermined service data in each service data block are configurable.

In an embodiment of the present application, the processing unit includes a processing module, an interception module, and a combination module, where the processing module is configured to process, based on the service logic, a plurality of pieces of the predetermined service data in a corresponding one of the service data blocks, one by one, to obtain a plurality of pieces of the target service data; the interception module is used for intercepting a plurality of pieces of target service data so that the plurality of pieces of target service data are not updated in the target database; the combination module is configured to form the target service block from a plurality of pieces of the target service data, and store the plurality of pieces of the target service data in the target service block in the JVM local cache. In the actual application process, in the process of carrying out piece-by-piece processing on a plurality of pieces of preset business data in corresponding business data blocks based on business logic, the obtained corresponding target business data can be updated into a target database at any time, that is, the target database is continuously connected and accessed, so that the connection and access pressures of the target database are higher. Therefore, in order to reduce the connection and access pressure of the target database, the target business data processed based on the business logic is intercepted, so that the target business data processed based on the business logic cannot be updated into the target database, and the connection and access pressure of the target database can be reduced.

In particular, in order to further and more simply avoid updating the target service data into the target database, in yet another embodiment of the present application, the interception module includes an interception sub-module, configured to intercept the MyBatis update operation, the MyBatis insert operation, and the MyBatis delete operation of the plurality of pieces of the target service data, so that the plurality of pieces of the target service data are not updated into the target database.

In another embodiment of the present application, the processing unit includes an updating module, configured to invoke at least a Java database connection (Java Database Connectivity, JDBC for short) to update the plurality of target service data in the JVM local cache into the target database in batches. In the embodiment, the target business data in the local buffer memory of the JVM is updated to the target database in batches through Java database connection, so that a plurality of business data in one target business block are updated to the target database in batches more simply and conveniently.

In order to further simply and conveniently update the multi-item target service data in one target service block to the target database in batches, that is, further simply and conveniently implement batch writing back of the multi-item target service data to the target database, in yet another embodiment of the present application, the update module includes an update sub-module, configured to invoke a batch update method, a batch insert method, and a batch delete method for the Java database connection, so as to update the target service data in the JVM local cache to the target database in batches.

In one embodiment of the present application, the blocking unit includes a receiving module and a blocking module, where the receiving module is configured to, when receiving a service data request, screen service data that meets the service data request from the target database to obtain a plurality of pieces of predetermined service data; the block module is used for blocking the plurality of pieces of preset business data according to preset rules to obtain a plurality of business data blocks, wherein the preset rules are rules for blocking the plurality of pieces of preset business data. In this embodiment, the plurality of pieces of predetermined service data are segmented according to the predetermined rule, so that the logic for partitioning the plurality of pieces of predetermined service data is simpler, the calculation amount is reduced, and the batch data updating device is lighter.

In still another embodiment of the present application, the updating apparatus further includes a clearing unit, configured to clear, after the target service data in the JVM local cache is updated to the target database in batches, a plurality of pieces of the target service data in the JVM local cache before closing the transaction, so as to ensure that the occupied JVM local cache can be released in time, and ensure that the robustness of the overall system is high.

The batch data updating device comprises a processor and a memory, wherein the block unit, the initialization unit, the processing unit, the repeating unit and the like are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize corresponding functions.

The processor includes a kernel, and the kernel fetches the corresponding program unit from the memory. The kernel can be provided with one or more than one kernel, and the problem of high cost of updating batch data in the prior art is solved by adjusting kernel parameters.

The memory may include volatile memory, random Access Memory (RAM), and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM), among other forms in computer readable media, the memory including at least one memory chip.

An embodiment of the present invention provides a computer-readable storage medium having stored thereon a program that, when executed by a processor, implements the above-described batch data updating method.

The embodiment of the invention provides a processor, which is used for running a program, wherein the program runs to execute the batch data updating method.

In an exemplary embodiment of the present application, a batch data updating system is also provided. The updating system comprises a UI interface, an updating device and a target database. The UI is used for sending the business data request to the updating device of the batch data; the batch data updating device is in communication connection with the UI interface and is used for executing any one of the batch data updating methods; the target database is in communication with the batch data updating device.

The batch data updating system comprises a UI interface, an updating device and a target database. The UI is used for sending the business data request to the updating device of the batch data; the batch data updating device is in communication connection with the UI interface and is used for executing any one of the batch data updating methods; the target database is in communication with the batch data updating device. In the above updating method, firstly, a plurality of pieces of predetermined service data are segmented to obtain a plurality of service data blocks; then, starting a transaction, and initializing a JVM local cache; then, based on service logic, processing a plurality of pieces of preset service data in a corresponding service data block to obtain a target service block, storing the multi-item target service data in the target service block in a JVM local cache, and updating the multi-item target service data in the JVM local cache to a target database in batches; finally, the initializing step and the processing step are sequentially executed at least once until the processing of all the service data blocks is completed. Compared with the prior art that Redis cache is adopted to update the multi-item label business data after business logic processing in batches, the method and the device are based on JVM local cache and transactions, namely the JVM local cache is limited in the transactions, and batch update is carried out on the multi-item label business data in the JVM local cache, so that the problem of data loss in the JVM local cache caused by difficulty in persistence of the JVM local cache can be avoided. Meanwhile, the JVM local cache is lighter and efficient, complex deployment and high deployment cost are not required, and network bandwidth occupation is not required, so that the method for updating the batch data is simpler and lower in cost, and the problem of higher cost for updating the batch data in the prior art is solved.

The embodiment of the invention provides equipment, which comprises a processor, a memory and a program stored in the memory and capable of running on the processor, wherein the processor realizes at least the following steps when executing the program:

The device herein may be a server, PC, PAD, cell phone, etc.

The present application also provides a computer program product adapted to perform a program initialized with at least the following method steps when executed on a data processing device:

In order that those skilled in the art can more clearly understand the technical solutions of the present application, the technical solutions and technical effects of the present application will be described below with reference to specific embodiments.

Examples

In an exemplary embodiment of the present application, a batch data update scheme is also provided, as shown in fig. 4. In this embodiment, the update scheme of batch data specifically includes the following steps:

step one: to support the configurability of batch update functionality. In this embodiment, a switch for turning on or off the batch update function may also be provided. For example, the configuration may be performed at a configuration center of the system, and a default configuration may be used. Default modes also include default on or default off. Of course, the default mode may also be configured according to the service function. For example, in a commercial banking system, batch transactions may be set on, online transactions may be set off, and so on;

step two: and acquiring a plurality of pieces of preset business data by a batch query mode (namely under the condition of receiving the business data sent by the UI interface). According to a preset rule, partitioning a plurality of pieces of preset service data to obtain one or more service data blocks;

Step three: starting a transaction, initializing the JVM local cache and acquiring whether to start a switch of a batch commit function;

step four: based on service logic, processing a plurality of pieces of preset service data in corresponding service data blocks one by one to obtain multi-item standard service data (namely processing data shown in fig. 4. In addition, the batch updating method of the application can also perform parallel operation, and only one service data block is used as an explanation in the embodiment);

step five: if the switch of the batch update function is turned on, intercepting the write-back operation of the target business data obtained based on business logic processing (namely intercepting the MyBatis update operation, the MyBatis insert operation and the MyBatis delete operation). Storing the multi-item label business data obtained after processing based on business logic into a JVM local cache (namely storing the multi-item label business data into the JVM local cache in fig. 4), and returning to successful processing;

step six: after the end of writing the data. If the switch of the batch updating function is started, the multi-item target business data to be updated is fetched from the JVM local cache. According to the write operation type, respectively calling a batch update method, a batch insertion method and a batch deletion method of the JDBC, and updating the multi-item mark business data processed based on the business logic to a target database in batches;

Step seven: after the multi-item target service data in one target service block is updated, the JVM local cache is emptied;

step eight: commit the transaction, i.e., close the transaction;

step nine: and when the processing of the plurality of preset business data in one business data block is finished, repeating the steps three to eight, namely, processing the plurality of preset business data in the next business data block until the processing of the plurality of preset business data in all business data blocks is finished.

By the batch data updating scheme, the following effects can be achieved:

1) By intercepting Mybatis database write operations (i.e., mybatis update operations, mybatis insert operations, and Mybatis delete operations). After the transaction, a batch updating method, a batch inserting method and a batch deleting method are carried out based on the JDBC, and the multi-item target service data in the target service block is updated to the target database. Therefore, the operand of one business data block connected with the target database can be reduced to 1, the interaction times with the target database are reduced, and the access pressure of the target database is reduced;

2) The JVM local cache has the characteristics of light weight and high efficiency. Batch updating is carried out based on the JVM local cache, and updating success is returned after the Mybatis database writing operation is intercepted, so that the system response rate is improved;

3) The life cycle of the JVM local cache is limited within the life cycle of the transaction, so that the influence of server fault cache invalidation on the application is avoided, and the robustness of the system is ensured;

4) In the scene of a banking system, if the daily final task involves the operation of a large number of single-node databases, under the scene of dividing the database into tables, the verification of technical fields such as the fields of dividing the database into tables and the optimistic lock version numbers is not needed when the single-table data is updated. The batch updating method based on the JDBC can avoid checking technical fields by Mybatis during single-table operation, and improves transaction efficiency under the condition of ensuring data correctness.

In the foregoing embodiments of the present invention, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed technology content may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units may be a logic function division, and there may be another division manner when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.

The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units described above, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the above-mentioned method of the various embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.

From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

1) In the batch data updating method, firstly, a plurality of pieces of preset business data are segmented to obtain a plurality of business data blocks; then, starting a transaction, and initializing a JVM local cache; then, based on service logic, processing a plurality of pieces of preset service data in a corresponding service data block to obtain a target service block, storing a plurality of pieces of target service data in the target service block in a JVM local cache, and updating a plurality of pieces of target service data in the JVM local cache to a target database in batches; finally, the initializing step and the processing step are sequentially executed at least once until the processing of all the service data blocks is completed. Compared with the prior art that Redis cache is adopted to update the multi-item label business data after business logic processing in batches, the method and the device are based on JVM local cache and transactions, namely the JVM local cache is limited in the transactions, and batch update is carried out on the multi-item label business data in the JVM local cache, so that the problem of data loss in the JVM local cache caused by difficulty in persistence of the JVM local cache can be avoided. Meanwhile, the JVM local cache is lighter and efficient, complex deployment and high deployment cost are not required, and network bandwidth occupation is not required, so that the method for updating the batch data is simpler and lower in cost, and the problem of higher cost for updating the batch data in the prior art is solved.

2) In the batch data updating device, the blocking unit is used for blocking a plurality of pieces of preset business data to obtain a plurality of business data blocks; the initialization unit is used for starting a transaction and initializing the JVM local cache; the processing unit is used for processing a plurality of pieces of preset business data in corresponding business data blocks based on business logic to obtain target business blocks, storing the multi-item target business data in the target business blocks in the JVM local cache, updating a plurality of pieces of target business data in the JVM local cache to the target database in batches, and the repeating unit is used for sequentially executing the initialization step and the processing step at least once until all business data blocks are processed. Compared with the prior art that Redis cache is adopted to update the multi-item label business data after business logic processing in batches, the method and the device are based on JVM local cache and transactions, namely the JVM local cache is limited in the transactions, and batch update is carried out on the multi-item label business data in the JVM local cache, so that the problem of data loss in the JVM local cache caused by difficulty in persistence of the JVM local cache can be avoided. Meanwhile, the JVM local cache is lighter and efficient, complex deployment and high deployment cost are not required, and network bandwidth occupation is not required, so that the method for updating the batch data is simpler and lower in cost, and the problem of higher cost for updating the batch data in the prior art is solved.

3) The batch data updating system comprises a UI interface, an updating device and a target database. The UI is used for sending the business data request to the updating device of the batch data; the batch data updating device is in communication connection with the UI interface and is used for executing any one of the batch data updating methods; the target database is in communication with the batch data updating device. In the above updating method, firstly, a plurality of pieces of predetermined service data are segmented to obtain a plurality of service data blocks; then, starting a transaction, and initializing a JVM local cache; then, based on service logic, processing a plurality of pieces of preset service data in a corresponding service data block to obtain a target service block, storing a plurality of pieces of target service data in the target service block in a JVM local cache, and updating a plurality of pieces of target service data in the JVM local cache to a target database in batches; finally, the initializing step and the processing step are sequentially executed at least once until the processing of all the service data blocks is completed. Compared with the prior art that Redis cache is adopted to update the multi-item label business data after business logic processing in batches, the method and the device are based on JVM local cache and transactions, namely the JVM local cache is limited in the transactions, and batch update is carried out on the multi-item label business data in the JVM local cache, so that the problem of data loss in the JVM local cache caused by difficulty in persistence of the JVM local cache can be avoided. Meanwhile, the JVM local cache is lighter and efficient, complex deployment and high deployment cost are not required, and network bandwidth occupation is not required, so that the method for updating the batch data is simpler and lower in cost, and the problem of higher cost for updating the batch data in the prior art is solved.

The foregoing description is only of the preferred embodiments of the present application and is not intended to limit the same, but rather, various modifications and variations may be made by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for updating batch data, comprising:

a blocking step, namely blocking a plurality of pieces of preset service data to obtain a plurality of service data blocks, wherein one service data block comprises a plurality of pieces of preset service data;

initializing, namely starting a transaction and initializing a JVM local cache;

processing, based on service logic, a plurality of pieces of preset service data in a corresponding service data block to obtain a target service block, storing the multi-item target service data in the target service block in the JVM local cache, and updating the plurality of pieces of target service data in the JVM local cache to a target database in batches and closing the transaction, wherein the target service data is service data obtained by processing the preset service data based on the service logic;

And repeating the initializing step and the processing step at least once in sequence until all the service data blocks are processed.

2. The updating method according to claim 1, wherein processing the plurality of pieces of the predetermined service data in the corresponding one of the service data blocks based on service logic to obtain a target service block, and storing the multi-item target service data in the target service block in the JVM local cache, comprises:

based on the service logic, processing a plurality of pieces of preset service data in a corresponding service data block piece by piece to obtain a plurality of pieces of target service data;

intercepting a plurality of pieces of target business data so that the plurality of pieces of target business data are not updated into the target database;

and forming the target service block by a plurality of pieces of target service data, and storing the plurality of pieces of target service data in the target service block in the JVM local cache.

3. The updating method according to claim 2, wherein intercepting the plurality of pieces of the target service data so that the plurality of pieces of the target service data are not updated into the target database comprises:

Intercepting MyBatis updating operation, myBatis inserting operation and MyBatis deleting operation of the target business data so that the target business data are not updated in the target database.

4. An updating method as defined in claim 1, wherein batch updating the plurality of target business data in the JVM local cache into a target database comprises:

and at least calling Java database connection to update a plurality of target business data in the JVM local cache to the target database in batches.

5. An updating method as defined in claim 4, wherein invoking at least a Java database connection to batch update the plurality of target business data in the JVM local cache into the target database comprises:

and calling a batch updating method, a batch inserting method and a batch deleting method of the Java database connection to update a plurality of target business data in the JVM local cache to the target database in batches.

6. The updating method according to any one of claims 1 to 5, wherein the partitioning of the plurality of pieces of predetermined service data to obtain a plurality of service data blocks includes:

Under the condition of receiving a service data request, screening service data meeting the service data request from the target database to obtain a plurality of pieces of preset service data;

and according to a preset rule, partitioning the plurality of pieces of preset business data to obtain a plurality of business data blocks, wherein the preset rule is a rule for partitioning the plurality of pieces of preset business data.

7. The method of updating according to any of claims 1 to 5, wherein after batch updating a plurality of the target business data in the JVM local cache into a target database, before closing the transaction, the method of updating further comprises:

and clearing a plurality of pieces of target business data in the JVM local cache.

8. An apparatus for updating batch data, comprising:

the block dividing unit is used for executing a block dividing step and dividing a plurality of pieces of preset service data to obtain a plurality of service data blocks, wherein one service data block comprises a plurality of pieces of preset service data;

the initialization unit is used for executing an initialization step, starting a transaction and initializing a JVM local cache;

The processing unit is used for executing the processing steps, processing a plurality of pieces of preset business data in a corresponding business data block based on business logic to obtain a target business block, storing a plurality of pieces of target business data in the target business block in the JVM local cache, updating a plurality of pieces of target business data in the JVM local cache to a target database in batches, and closing the transaction, wherein the target business data is business data after processing the preset business data based on the business logic;

and the repeating unit is used for executing the repeating steps, and sequentially repeating the initializing step and the processing step at least once until the processing of all the service data blocks is completed.

9. A computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored program, wherein the program performs the batch data updating method of any one of claims 1 to 7.

10. A batch data update system, comprising:

UI interface for transmitting business data request to batch data updating device

The batch data updating device is in communication connection with the UI interface and is used for executing the batch data updating method according to any one of claims 1 to 7;

and the target database is communicated with the batch data updating device.