WO2020224216A1

WO2020224216A1 - Bulk data processing method and apparatus, and computer device and storage medium

Info

Publication number: WO2020224216A1
Application number: PCT/CN2019/117745
Authority: WO
Inventors: 孙强; 唐龙
Original assignee: 平安科技（深圳）有限公司
Priority date: 2019-05-05
Filing date: 2019-11-13
Publication date: 2020-11-12
Also published as: CN110263048A

Abstract

Disclosed are a bulk data processing method and apparatus, and a computer device and a storage medium. The present application is applied to the field of data updating of big data. The method comprises: acquiring all batches of data to be processed from a preset database, and allocating a batch number to each batch of data to be processed; taking a batch of data to be processed that has one of the batch numbers as a target batch of data, adding an exclusive lock to the target batch of data, and processing the target batch of data; if the processing of the target batch of data is completed, acquiring an initial preset field value in the target batch of data, and submitting the target batch of data to the preset database according to the initial preset field value in order to release the exclusive lock; and taking a batch of data to be processed that has the next batch number as a target batch of data according to the batch numbers, and processing same until the processing of all the batches of data to be processed that have the batch numbers is completed.

Description

Mass data processing method, device, computer equipment and storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 201910367480.8, and the application name is "mass data processing methods, devices, computer equipment and storage media" on May 5, 2019. The reference is incorporated in this application.

Technical field

This application relates to the field of data processing technology, and in particular to a method, device, computer equipment, and storage medium for processing mass data.

Background technique

With the development of economy and technology, data is growing explosively, huge amounts of data are generated every day, and huge amounts of data need to be processed every day. Especially in the field of collection, collection data has the characteristics of large orders of magnitude and very frequent updates and changes. The process of dynamically updated collection data through the strategy engine can effectively control risks. Therefore, the existing collection system needs to process large quantities of data every day. However, the reading and writing of large quantities of data in the database takes a long time and puts a lot of pressure on the server, which easily leads to repeated data capture and omission of data processing. .

Summary of the invention

The embodiments of the present application provide a method, device, computer device, and storage medium for processing large quantities of data, aiming to solve the problem of long time-consuming processing of large quantities of data, repeated capture and missing processing.

In the first aspect, an embodiment of the present application provides a method for processing large batch data, which includes: acquiring all batch data to be processed from a preset database and assigning a batch number to each batch of data to be processed; One of the batch numbers of the batch data to be processed is used as the target batch data, an exclusive lock is added to the target batch data and the target batch data is processed; if the target batch data processing is completed, all the data are obtained The initial preset field value in the target batch data and submit the target batch data to the preset database according to the initial preset field value to release the exclusive lock; and download according to the batch number The to-be-processed batch data of a batch number is processed as the target batch data until all the to-be-processed batch data of all batch numbers are processed.

In the second aspect, an embodiment of the present application also provides a mass data processing device, which includes: a distribution unit, configured to obtain all batch data to be processed from a preset database and Allocate batch numbers; an exclusive unit for using the batch data to be processed of one of the batch numbers as target batch data, add an exclusive lock to the target batch data, and process the target batch data; release Unit, configured to obtain the initial preset field value in the target batch data if the target batch data processing is completed, and submit the target batch data to the preset according to the initial preset field value Database to release the exclusive lock; a processing unit for processing the batch data to be processed with the next batch number as the target batch data according to the batch number until the batch number to be processed for all batch numbers All batch data is processed.

In a third aspect, an embodiment of the present application also provides a computer device, which includes a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the A computer program to execute the following steps: obtain all batch data to be processed from a preset database and assign a batch number to each batch data to be processed; and assign the batch data to be processed with one of the batch numbers As the target batch data, add an exclusive lock to the target batch data and process the target batch data; if the target batch data processing is completed, obtain the initial preset field value in the target batch data and Submit the target batch data to the preset database according to the initial preset field value to release the exclusive lock; use the batch data to be processed of the next batch number as the batch number according to the batch number The target batch data is processed until the to-be-processed batch data of all batch numbers are all processed.

In a fourth aspect, the embodiments of the present application also provide a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the following steps: Obtain all the batch data to be processed from the preset database and assign a batch number to each batch data to be processed; take the batch data to be processed with one of the batch numbers as the target batch data, and Add an exclusive lock to the target batch data and process the target batch data; if the target batch data processing is completed, obtain the initial preset field value in the target batch data and change the value according to the initial preset field value The target batch data is submitted to the preset database to release the exclusive lock; the batch data to be processed of the next batch number is processed as the target batch data according to the batch number until all batches are processed The processing of the batch data to be processed of the minor number is completed.

Description of the drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following will briefly introduce the drawings needed in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present application. Ordinary technicians can obtain other drawings based on these drawings without creative work.

FIG. 1 is a schematic flow chart of a method for processing large batch data provided by an embodiment of the application;

FIG. 2 is a schematic diagram of a sub-flow of a method for processing large quantities of data provided by an embodiment of the application;

FIG. 3 is a schematic diagram of a sub-flow of a method for processing large batches of data provided by an embodiment of the application;

FIG. 4 is a schematic diagram of a sub-flow of a method for processing mass data according to an embodiment of the application;

FIG. 5 is a schematic diagram of a sub-flow of a method for processing a large batch of data provided by an embodiment of the application;

FIG. 6 is a schematic block diagram of a mass data processing device provided by an embodiment of the application;

FIG. 7 is a schematic block diagram of specific units of a mass data processing device provided by an embodiment of the application; and

FIG. 8 is a schematic block diagram of a computer device provided by an embodiment of the application.

Detailed ways

The technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

It should be understood that when used in this specification and the appended claims, the terms "including" and "including" indicate the existence of the described features, wholes, steps, operations, elements and/or components, but do not exclude one or The existence or addition of multiple other features, wholes, steps, operations, elements, components, and/or collections thereof.

It should also be understood that the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in the specification of this application and the appended claims, unless the context clearly indicates other circumstances, the singular forms "a", "an" and "the" are intended to include plural forms.

It should be further understood that the term "and/or" used in the specification and appended claims of this application refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations .

Please refer to Fig. 1. Fig. 1 is a schematic flow chart of a method for processing large batch data according to an embodiment of the application. The mass data processing method can be applied to the collection system and implemented in the server. The collection system is an information system used by a lending company to control loan data. The system includes a large amount of collection-related data. With the change of daily collection data, the collection system needs to process large quantities of collection data every day. Existing databases usually use optimistic locking to manipulate data when processing data with fewer write operations. When operating data through optimistic locking, other threads will not modify the data by default, so the data will not be locked, just before updating the data Determine whether other threads have modified the data during this period. Using optimistic locking to manipulate data is highly efficient, but it produces more concurrency conflicts and reduces performance; existing databases usually use pessimistic locks to manipulate data when processing data with more write operations , When processing data through pessimistic lock, other threads will modify the data by default. Therefore, the data is locked before the data is operated. After the data is locked, other threads cannot operate on it. The use of pessimistic lock to operate data has less concurrency conflicts, but on Locking is expensive and resource intensive, and threads often need to be suspended and wait, which reduces performance. For the collection system, a large amount of collection data needs to be processed every day, and the collection data is not only large in size, but also frequently updated. Therefore, by adopting the mass data processing method of the present application, through the combination of optimistic locking and pessimistic locking, mass data can be processed quickly and efficiently while ensuring that the data will not be repeatedly fetched or omitted.

FIG. 1 is a schematic flowchart of a method for processing large batches of data provided by an embodiment of the present application. As shown in the figure, the method includes the following steps S110-S140.

S110: Obtain all batch data to be processed from a preset database and assign a batch number to each batch data to be processed.

In one embodiment, the preset database refers to the database of the collection system, and the collection data of the collection system is stored in the preset database. The batch data to be processed refers to the collection data that needs to be processed in batches. The batches of the batch data to be processed are pre-divided according to the data size. For example, the data size of the batch data to be processed is 100G in total, with 1G A batch of data to be processed is divided into 100 batches. The batch number refers to the number assigned to the batch of data to be processed, and each batch of data to be processed is assigned a batch number, for example, Data processing001. The batch number is used to process the batch data to be processed, which can ensure that no processing is missed.

In an embodiment, as shown in FIG. 2, the step S110 may include steps S111-S113.

S111. Obtain all batch data to be processed from a preset database.

S111. Assign a batch number according to the batch quantity of the batch data to be processed.

S112: Sort the batch data to be processed from small to large according to the batch number.

In one embodiment, first obtain all the batch data to be processed from the preset database, and then assign a batch to each batch of batch data to be processed according to the batch quantity of the obtained batch data to be processed For example, if the batch number of the batch data to be processed is 100, then a batch number is assigned to the 100 batches of batch data from 0 to 100, and finally according to the batch number in ascending order Sort in big order. When processing the batch data to be processed, the processing starts from the smallest batch number according to the sorting order until the processing batch data of the largest batch number is processed, so as to ensure that no processing is missed.

S120: Use the batch data to be processed of one of the batch numbers as target batch data, add an exclusive lock to the target batch data, and process the target batch data.

In one embodiment, the exclusive lock is a locking mechanism in the pessimistic lock. The exclusive lock is used to lock the data during the entire data processing process, and other processes cannot read or write the data. In order to avoid inconsistencies in data reading and writing caused by other processes reading and writing the data being processed during data processing, the batch data to be processed with one of the batch numbers is used as the target batch data, and an exclusive lock mechanism is adopted. Lock the target batch of data to prevent other processes from reading and writing the processed target batch of data, so as to maintain the consistency of data read and write.

In an embodiment, as shown in FIG. 3, the step S120 may include steps S121-S124.

S121: Use the batch data to be processed of one of the batch numbers as target batch data.

S122. Call the exclusive lock mechanism of the preset database to add an exclusive lock to the target batch of data.

S123. Process the target batch data to which the exclusive lock has been added.

S124: Add a processing mark to the processed target batch data.

In one embodiment, the exclusive lock is a lock mechanism of the database itself. Therefore, when the exclusive lock is added to the target batch data, the exclusive lock mechanism of the preset database is directly called to lock the target batch data, specifically , The target batch data can be locked by adding for update in the execution statement for processing the target batch data. After the lock is locked, the target batch data is processed. The processing performed on the target batch data includes data addition, deletion, and modification. For example, yesterday, Zhang San added a loan of 50,000 yuan, and Li Si repaid 1 Ten thousand yuan, Wang Wu’s loan record has reached five years, so the target batch data processed in batches today includes the addition of Zhang San’s new loan of 50,000 yuan in the preset database, and the reduction of Li Si’s loan amount of 10,000 yuan. Delete the overdue Wang Wu's loan record. After processing the target batch of data, a processing mark is added to indicate that the processing of the batch of data to be processed has been completed.

S130. If the target batch data processing is completed, obtain an initial preset field value in the target batch data and submit the target batch data to the preset database according to the initial preset field value. Release the exclusive lock.

In one embodiment, the initial preset field value refers to the preset field value when the target batch of data is first obtained from the preset database, and each batch of data to be processed includes a preset field value. The preset field value can be a version number (version) or a timestamp (timestamp). If the preset field value is a version number, the specific form is a numeric field, such as 1, and the version number is incremented by one after the data update is completed, for example, the data with the version number of 1 is updated to 2; if the field value is preset It is a timestamp, and its specific form is a field of time type, for example, 2018.11.20, 00:00. The timestamp is updated every time the data update is completed. Optimistic locking is achieved through initial preset field values. When acquiring the target batch data, other threads will not process the target batch data by default, but when the target batch data is processed and submitted to the preset database The optimistic lock is triggered, and it is judged according to the initial preset field value whether other threads have modified the target batch data during the period from acquiring the target batch data to the completion of processing the target batch data. If other threads do not modify the target batch data during this period, submit the processed target batch data to the preset database to release the exclusive lock, and other threads can immediately modify the target batch The data was processed.

In an embodiment, as shown in FIG. 3, the step S130 may include steps S131-S134.

S131. Obtain an initial preset field value from the target batch data.

S132. Obtain the current preset field value of the target batch data from the preset database.

S133. Compare the initial preset field value with the current preset field value.

S134: If the initial preset field value is the same as the current preset field value, submit the target batch data to the preset database to release the exclusive lock.

In one embodiment, since the preset field value is a field value added to the target batch data, the initial preset field value can be obtained from the target batch data. Then, the current preset field value in the target batch data of the batch is read from the preset database. Then compare the initial preset field value of the target batch of data with the current preset field value. If the two are the same, it means that other threads are in the period from acquiring the target batch of data to processing the target batch of data. The target batch data has not been modified. Since the exclusive lock mechanism of the preset database is automatically released when the database transaction is submitted, finally the processed target batch data is submitted to the preset database for update and then released to release the target batch data With the exclusive lock, other processes can immediately perform new processing on the target batch of data. Wherein, if the initial preset field value of the target batch of data is not the same as the current preset field value, it means that during the period from the acquisition of the target batch of data to the completion of the processing of the target batch of data by other threads, The data has been modified, and the error message is returned to the collection system at this time for further processing by the collection system.

S140: Process the batch data to be processed of the next batch number as target batch data according to the batch number until all the batch data to be processed of all batch numbers are processed.

In an embodiment, the processing starts from the batch data to be processed with the smallest batch number, and the batch data to be processed with the smallest batch number is processed, and then the batch to be processed with the next batch number is processed. The secondary data will be processed one by one in the descending order of the assigned batch number. For example, the minimum batch number is 1. After the processing of the batch data to be processed with the batch number 1 is completed, the batch data to be processed with the batch number 2 is processed until all batch numbers are The batch data to be processed is all processed. According to the processing sequence of the batch number from small to large, it can be ensured that all the batch data to be processed can be processed to prevent the omission of processing.

In an embodiment, as shown in FIG. 4, the step S140 may include steps S141-S143.

S141. Obtain the batch data to be processed of the next batch number according to the batch number.

S142: Determine whether the processing mark exists in the batch of data to be processed.

S143: If the processing mark exists in the batch data to be processed, the batch data to be processed with the next batch number is used as the target batch data for processing.

In one embodiment, before processing the to-be-processed batch data of the next batch number, it is necessary to judge the currently processed batch-to-be-processed data, and determine whether the currently processed batch data Whether there is a processing mark, if there is a processing mark in the current batch of data to be processed, it means that the processing of the current batch of data to be processed has been completed, and the processing of the next batch of the batch of data to be processed starts, namely Return to step S120 to start processing the batch data to be processed with the next batch number, until all the batch data to be processed are completely processed. Wherein, if there is no processing mark in the current batch of data to be processed, it means that the current batch of data to be processed is abnormal during processing, for example, a certain piece of data is missing or processed incorrectly, then return The error message is sent to the collection system, and the collection system traces the batch data to be processed for further processing.

When processing large batches of collection data, this application first uses optimistic locking to obtain the batch data to be processed from the preset database, first does not lock the batch data to be processed, and other threads can still lock the The batch data to be processed is operated. Then according to the order of the batch number, the batch data to be processed with the smallest batch number is processed first, and the batch data to be processed with the smallest batch number is used as the target batch data. When adding an exclusive lock to the target batch data of the batch number, other threads cannot operate on the target batch data of the batch number at this time, but other threads can still perform operations on other batch numbers at this time. Of the batch of data to be processed. Then, after the target batch data of the batch number is processed, the optimistic lock is triggered when submitted to the preset database to determine whether the target batch data of the batch number has been modified by other threads during the process. If it has not been modified by other threads during this period, submit the processed batch number of the target batch data to the preset database for update, and at the same time release the exclusive lock of the target batch data of the batch number , Other threads can immediately modify the target batch data of the batch number. Therefore, the collection data processed by the mass data processing method of the present application can quickly capture mass collection data, and has little impact on other threads when processing collection data, and other threads can quickly perform new processing and ensure data There is no repeated fetching and missing processing, which greatly improves the processing speed and efficiency of large quantities of data.

The embodiment of the application shows a data processing method, by acquiring all batch data to be processed from a preset database and assigning a batch number to each batch of data to be processed; The batch data to be processed is used as the target batch data, an exclusive lock is added to the target batch data and the target batch data is processed; if the target batch data processing is completed, the initial batch data in the target batch data is obtained Preset field values and submit the target batch data to the preset database according to the initial preset field values to release the exclusive lock; according to the batch number, the pending batch number of the next batch number The processed batch data is processed as the target batch data until the processing of the batch data to be processed of all batch numbers is completed, which can ensure that the data will not be repeatedly fetched and missed processing, and increase the processing speed of large batch data.

FIG. 6 is a schematic block diagram of a mass data processing apparatus 200 provided by an embodiment of the present application. As shown in FIG. 6, corresponding to the above mass data processing method, the present application also provides a mass data processing device 200. The mass data processing device 200 includes a unit for executing the above mass data processing method, and the device can be configured in a server. Specifically, referring to FIG. 6, the mass data processing apparatus 200 includes: an allocation unit 210, an exclusive unit 220, a release unit 230 and a processing unit 240.

The allocation unit 210 is configured to obtain all batch data to be processed from a preset database and allocate a batch number to each batch data to be processed.

In an embodiment, as shown in FIG. 7, the allocation unit 210 includes: an acquisition subunit 211, an allocation subunit 212, and a sorting unit 213.

The obtaining subunit 211 is used to obtain all batch data to be processed from a preset database.

The allocation subunit 212 is configured to allocate batch numbers according to the batch quantity of the batch data to be processed.

The sorting unit 213 is configured to sort the batch data to be processed from small to large according to the batch number.

The exclusive unit 220 is configured to use the batch data to be processed with one of the batch numbers as target batch data, add an exclusive lock to the target batch data, and process the target batch data.

In an embodiment, as shown in FIG. 7, the exclusive unit 220 includes a target unit 221, a calling unit 222, an adding unit 223 and a marking unit 224.

The target unit 221 is configured to use the batch data to be processed of one of the batch numbers as target batch data.

The calling unit 222 is configured to call the exclusive lock mechanism of the preset database to add an exclusive lock to the target batch of data.

The adding unit 223 is configured to process the target batch data to which the exclusive lock has been added.

The marking unit 224 is used to add a processing mark to the processed target batch data.

The releasing unit 230 is configured to obtain the initial preset field value in the target batch data and submit the target batch data to the target batch data according to the initial preset field value if the processing of the target batch data is completed. The database is preset to release the exclusive lock.

In an embodiment, as shown in FIG. 7, the releasing unit 230 includes: a first acquiring unit 231, a second acquiring unit 232, a comparing unit 233 and a submitting unit 234.

The first obtaining unit 231 is configured to obtain an initial preset field value from the target batch data;

The second obtaining unit 232 is configured to obtain the current preset field value of the target batch data from the preset database;

The comparison unit 233 is configured to compare the initial preset field value with the current preset field value;

The submitting unit 234 is configured to submit the target batch data to the preset database to release the exclusive lock if the initial preset field value is the same as the current preset field value.

The processing unit 240 is configured to process the to-be-processed batch data of the next batch number as target batch data according to the batch number until all the to-be-processed batch data of all batch numbers are processed.

In an embodiment, as shown in FIG. 7, the processing unit 240 includes: a third acquisition unit 241, a judgment unit 242 and a loop processing unit 243.

The third obtaining unit 241 is configured to obtain the batch data to be processed of the next batch number according to the batch number;

The determining unit 242 is configured to determine whether the processing mark exists in the batch of data to be processed;

The loop processing unit 243 is configured to process the to-be-processed batch data of the next batch number as the target batch data if the processing mark exists in the to-be-processed batch data.

It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned mass data processing apparatus 200 and each unit can refer to the corresponding description in the foregoing method embodiment. For the convenience and conciseness of the description, This will not be repeated here.

The above-mentioned mass data processing apparatus may be implemented in the form of a computer program, and the computer program may be run on a computer device as shown in FIG. 8.

Please refer to FIG. 8. FIG. 8 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 may be a server. The server may be an independent server or a server cluster composed of multiple servers.

Referring to FIG. 8, the computer device 500 includes a processor 502, a memory, and a network interface 505 connected through a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.

The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions, and when the program instructions are executed, the processor 502 can execute a method for processing mass data.

The processor 502 is used to provide calculation and control capabilities to support the operation of the entire computer device 500.

The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a large-scale data processing method.

The network interface 505 is used for network communication with other devices. Those skilled in the art can understand that the structure shown in FIG. 8 is only a block diagram of part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device 500 to which the solution of the present application is applied. The specific computer device 500 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

Wherein, the processor 502 is configured to run a computer program 5032 stored in a memory, so as to implement the mass data processing method in the embodiment of the present application.

It should be understood that, in this embodiment of the application, the processor 502 may be a central processing unit (Central Processing Unit, CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSPs), Application Specific Integrated Circuit (ASIC), ready-made programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the foregoing embodiments can be implemented by computer programs instructing relevant hardware. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the foregoing method embodiments.

Therefore, this application also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the mass data processing method described in the above embodiments.

The storage medium may be a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a magnetic disk or an optical disk, and other computer-readable storage media that can store program codes.

A person of ordinary skill in the art may realize that the units and algorithm steps of the examples described in the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two, in order to clearly illustrate the hardware and software Interchangeability. In the above description, the composition and steps of each example have been generally described in terms of function. Whether these functions are executed by hardware or software depends on the specific application and design constraint conditions of the technical solution. Professionals and technicians can use different methods for each specific application to implement the described functions, but such implementation should not be considered beyond the scope of this application.

In the several embodiments provided in this application, it should be understood that the disclosed device and method may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be omitted or not implemented.

The steps in the method of the embodiment of the present application can be adjusted, merged, and deleted in order according to actual needs. The units in the devices in the embodiments of the present application may be combined, divided, and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this application is essentially or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium It includes several instructions to make a computer device (which may be a personal computer, a terminal, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application.

The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Anyone familiar with the technical field can easily think of various equivalents within the technical scope disclosed in this application. Modifications or replacements, these modifications or replacements shall be covered within the protection scope of this application. Therefore, the protection scope of this application shall be subject to the protection scope of the claims.

Claims

A method for processing large quantities of data, including:

Obtain all batch data to be processed from the preset database and assign a batch number to each batch data to be processed;

Taking the batch data to be processed of one of the batch numbers as target batch data, adding an exclusive lock to the target batch data, and processing the target batch data;

If the target batch data processing is completed, obtain the initial preset field value in the target batch data and submit the target batch data to the preset database according to the initial preset field value to release all State exclusive lock;

The batch data to be processed of the next batch number is processed as the target batch data according to the batch number until all the batch data to be processed of all batch numbers are processed.
The method for processing large batch data according to claim 1, wherein said acquiring all batch data to be processed from a preset database and assigning a batch number to each batch data to be processed comprises:

Obtain all batch data to be processed from the preset database;

Allocating batch numbers according to the batch quantity of the batch data to be processed;

Sort the batch data to be processed according to the batch number from small to large.
The method according to claim 1, wherein the batch data to be processed with one of the batch numbers is used as the target batch data, an exclusive lock is added to the target batch data, and all the batch data is processed State the target batch data, including:

Use the batch data to be processed of one of the batch numbers as the target batch data;

Calling the exclusive lock mechanism of the preset database to add an exclusive lock to the target batch of data;

Processing the target batch data to which the exclusive lock has been added;

Add a processing mark to the processed target batch data.
The method for processing mass data according to claim 1, wherein said acquiring an initial preset field value in said target batch data and submitting said target batch data to said target batch data according to said initial preset field value The preset database to release the exclusive lock includes:

Obtaining initial preset field values from the target batch data;

Acquiring the current preset field value of the target batch data from the preset database;

Comparing the initial preset field value with the current preset field value;

If the initial preset field value is the same as the current preset field value, submit the target batch data to the preset database to release the exclusive lock.
The mass data processing method according to claim 3, wherein the batch data to be processed of the next batch number is processed as the target batch data according to the batch number until all batch numbers are The processing of the batch data to be processed is completed, including:

Acquiring the batch data to be processed of the next batch number according to the batch number;

Determine whether the processing mark exists in the batch of data to be processed;

If the processing mark exists in the batch data to be processed, the batch data to be processed with the next batch number is used as the target batch data for processing.
The method for processing mass data according to claim 4, wherein the initial preset field value is a version number.
The method for processing mass data according to claim 4, wherein the initial preset field value is a timestamp.
A mass data processing device, including:

The allocation unit is configured to obtain all batch data to be processed from the preset database and allocate a batch number to each batch data to be processed;

An exclusive unit, configured to use the batch data to be processed of one of the batch numbers as target batch data, add an exclusive lock to the target batch data, and process the target batch data;

The releasing unit is configured to, if the processing of the target batch data is completed, obtain the initial preset field value in the target batch data and submit the target batch data to the preset according to the initial preset field value Set up a database to release the exclusive lock;

The processing unit is configured to process the to-be-processed batch data of the next batch number as target batch data according to the batch number until all the to-be-processed batch data of all batch numbers are processed.
The mass data processing device according to claim 8, wherein the allocating unit comprises:

The acquiring subunit is used to acquire all the batch data to be processed from the preset database;

An allocation sub-unit for allocating a batch number according to the batch quantity of the batch data to be processed;

The sorting unit is used to sort the batch data to be processed from small to large according to the batch number.
The mass data processing device according to claim 8, wherein the exclusive unit comprises:

The target unit is used to use the batch data to be processed of one of the batch numbers as target batch data;

A calling unit for calling an exclusive lock mechanism of the preset database to add an exclusive lock to the target batch of data;

The adding unit is used to process the target batch data to which the exclusive lock has been added;

The marking unit is used to add a processing mark to the processed target batch data.
A computer device includes a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to perform the following steps:

Obtain all batch data to be processed from the preset database and assign a batch number to each batch data to be processed;

Taking the batch data to be processed of one of the batch numbers as target batch data, adding an exclusive lock to the target batch data, and processing the target batch data;

If the target batch data processing is completed, obtain the initial preset field value in the target batch data and submit the target batch data to the preset database according to the initial preset field value to release all State exclusive lock;

The batch data to be processed of the next batch number is processed as the target batch data according to the batch number until all the batch data to be processed of all batch numbers are processed.
11. The computer device according to claim 11, wherein said acquiring all batch data to be processed from a preset database and assigning a batch number to each batch data to be processed comprises:

Obtain all batch data to be processed from the preset database;

Allocating batch numbers according to the batch quantity of the batch data to be processed;

Sort the batch data to be processed according to the batch number from small to large.
The computer device according to claim 11, wherein the batch data to be processed of one of the batch numbers is used as target batch data, an exclusive lock is added to the target batch data, and the target batch is processed Secondary data, including:

Use the batch data to be processed of one of the batch numbers as the target batch data;

Calling the exclusive lock mechanism of the preset database to add an exclusive lock to the target batch of data;

Processing the target batch data to which the exclusive lock has been added;

Add a processing mark to the processed target batch data.
The computer device according to claim 11, wherein said acquiring the initial preset field value in the target batch data and submitting the target batch data to the preset according to the initial preset field value The database to release the exclusive lock, including:

Obtaining initial preset field values from the target batch data;

Acquiring the current preset field value of the target batch data from the preset database;

Comparing the initial preset field value with the current preset field value;

If the initial preset field value is the same as the current preset field value, submit the target batch data to the preset database to release the exclusive lock.
The computer device according to claim 13, wherein the batch data of the next batch number to be processed is processed as target batch data according to the batch number until the batch number of all batch numbers is to be processed. All processing batch data is completed, including:

Acquiring the batch data to be processed of the next batch number according to the batch number;

Determine whether the processing mark exists in the batch of data to be processed;

If the processing mark exists in the batch data to be processed, the batch data to be processed with the next batch number is used as the target batch data for processing.
The computer device according to claim 14, wherein the initial preset field value is a version number.
The computer device according to claim 14, wherein the initial preset field value is a timestamp.
A computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the processor executes the following steps:

Obtain all batch data to be processed from the preset database and assign a batch number to each batch data to be processed;

Taking the batch data to be processed of one of the batch numbers as target batch data, adding an exclusive lock to the target batch data, and processing the target batch data;

If the target batch data processing is completed, obtain the initial preset field value in the target batch data and submit the target batch data to the preset database according to the initial preset field value to release all State exclusive lock;

The batch data to be processed of the next batch number is processed as the target batch data according to the batch number until all the batch data to be processed of all batch numbers are processed.
18. The computer-readable storage medium according to claim 18, wherein the step of acquiring all batch data to be processed from a preset database and assigning a batch number to each batch data to be processed comprises:

Obtain all batch data to be processed from the preset database;

Allocating batch numbers according to the batch quantity of the batch data to be processed;

Sort the batch data to be processed according to the batch number from small to large.
The computer-readable storage medium according to claim 18, wherein the batch data to be processed of one of the batch numbers is used as target batch data, an exclusive lock is added to the target batch data, and all the batch data is processed The steps to describe the target batch data include:

Use the batch data to be processed of one of the batch numbers as the target batch data;

Calling the exclusive lock mechanism of the preset database to add an exclusive lock to the target batch of data;

Processing the target batch data to which the exclusive lock has been added;

Add a processing mark to the processed target batch data.