CN110263048B - Method and device for processing large-batch data, computer equipment and storage medium - Google Patents
Method and device for processing large-batch data, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110263048B CN110263048B CN201910367480.8A CN201910367480A CN110263048B CN 110263048 B CN110263048 B CN 110263048B CN 201910367480 A CN201910367480 A CN 201910367480A CN 110263048 B CN110263048 B CN 110263048B
- Authority
- CN
- China
- Prior art keywords
- data
- batch
- processed
- batch data
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 133
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000004590 computer program Methods 0.000 claims description 17
- 238000012163 sequencing technique Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000003203 everyday effect Effects 0.000 description 4
- 238000003672 processing method Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000002354 daily effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- General Factory Administration (AREA)
Abstract
The embodiment of the invention discloses a method, a device, computer equipment and a storage medium for processing mass data. The method and the device are applied to the field of data updating in big data. The method comprises the following steps: acquiring all batch data to be processed from a preset database and distributing batch numbers to each batch data to be processed; taking the batch data to be processed of one batch number as target batch data, adding an exclusive lock to the target batch data and processing the target batch data; if the target batch data is processed, acquiring an initial preset field value in the target batch data, and submitting the target batch data to a preset database according to the initial preset field value so as to release the exclusive lock; and processing the batch data to be processed of the next batch number as target batch data according to the batch number until all the batch data to be processed of all the batch numbers are processed completely. By implementing the method of the embodiment of the invention, the data can be ensured not to be repeatedly grabbed and not to be subjected to missing processing, and the processing speed of a large amount of data is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a method and apparatus for processing large-scale data, a computer device, and a storage medium.
Background
With the development of economy and technology, data is in explosive growth, and huge amounts of data are generated every day, and the huge amounts of data need to be processed every day. Particularly in the field of collection, the collection data has the characteristics of large order of magnitude and very frequent update change, and the risk can be effectively controlled by processing the dynamically updated collection data through a policy engine. Therefore, the existing collection system needs to process a large amount of data every day, however, the time for reading and writing the large amount of data in the database is long, the pressure on the server is high, and the situation of repeatedly capturing the data and processing the data is easy to occur.
Disclosure of Invention
The embodiment of the invention provides a method, a device, computer equipment and a storage medium for processing large-batch data, which aim to solve the problems of long time consumption, repeated grabbing and missing processing in processing the large-batch data.
In a first aspect, an embodiment of the present invention provides a method for processing bulk data, including: acquiring all batch data to be processed from a preset database and distributing batch numbers to each batch data to be processed; taking the batch data to be processed of one batch number as target batch data, adding an exclusive lock to the target batch data and processing the target batch data; if the target batch data is processed, acquiring an initial preset field value in the target batch data and submitting the target batch data to the preset database according to the initial preset field value so as to release the exclusive lock; and processing the batch data to be processed of the next batch number as target batch data according to the batch number until all the batch data to be processed of all the batch numbers are processed.
In a second aspect, an embodiment of the present invention further provides a mass data processing apparatus, including: the distribution unit is used for acquiring all batch data to be processed from a preset database and distributing batch numbers to each batch data to be processed; an exclusive unit for taking the batch data to be processed of one batch number as target batch data, adding an exclusive lock to the target batch data and processing the target batch data; the release unit is used for acquiring an initial preset field value in the target batch data and submitting the target batch data to the preset database according to the initial preset field value so as to release the exclusive lock if the target batch data is processed; and the processing unit is used for processing the batch data to be processed of the next batch number as target batch data according to the batch number until all the batch data to be processed of all the batch numbers are processed completely.
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the method when executing the computer program.
In a fourth aspect, embodiments of the present invention also provide a computer readable storage medium storing a computer program which, when executed by a processor, implements the above method.
The embodiment of the invention provides a method and a device for processing mass data, computer equipment and a storage medium. Wherein the method comprises the following steps: acquiring all batch data to be processed from a preset database and distributing batch numbers to each batch data to be processed; taking the batch data to be processed of one batch number as target batch data, adding an exclusive lock to the target batch data and processing the target batch data; if the target batch data is processed, acquiring an initial preset field value in the target batch data and submitting the target batch data to the preset database according to the initial preset field value so as to release the exclusive lock; and processing the batch data to be processed of the next batch number as target batch data according to the batch number until all the batch data to be processed of all the batch numbers are processed. According to the embodiment of the invention, the batch number is allocated to the batch data to be processed and the exclusive lock is added, so that the data can be ensured not to be repeatedly grabbed and not to be leaked, and the processing speed of the large batch of data is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for processing bulk data according to an embodiment of the present invention;
FIG. 2 is a schematic sub-flowchart of a method for processing bulk data according to an embodiment of the present invention;
FIG. 3 is a schematic flow chart of a method for processing large-scale data according to an embodiment of the present invention;
FIG. 4 is a schematic flow chart of a method for processing large batch data according to an embodiment of the present invention;
FIG. 5 is a schematic sub-flowchart of a method for processing bulk data according to an embodiment of the present invention;
FIG. 6 is a schematic block diagram of a mass data processing apparatus provided by an embodiment of the present invention;
FIG. 7 is a schematic block diagram of specific units of a mass data processing apparatus provided by an embodiment of the present invention; and
Fig. 8 is a schematic block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be understood that the terms "comprises" and "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
Referring to fig. 1, fig. 1 is a schematic flowchart of a method for processing bulk data according to an embodiment of the present invention. The method for processing the large batch of data can be applied to a collection accelerating system and is realized in a server. The system is an information system for controlling loan data of a loan company, and comprises a large amount of data related to the loan, and the system is required to process a large amount of the data along with the change of the daily data. When the existing database processes data with fewer writing operations, the data is usually operated by adopting an optimistic lock, other threads are defaulted to not modify the data when the data is operated by the optimistic lock, so the data cannot be locked, only whether the other threads modify the data in the period is judged before the data is updated, the data efficiency is high by adopting the optimistic lock to operate the data, but the generated concurrency conflicts are more, and the performance is reduced; when the existing database processes data with more writing operations, the data is usually operated by pessimistic locks, other threads default to modify the data when the data is processed by the pessimistic locks, so that the data is locked before the data is operated, the other threads of the locked data cannot operate the data, the pessimistic locks are adopted to operate the data and have less concurrent conflict, but the locking cost is high, resources are consumed, and the threads always need to be hung to wait, so that the performance is reduced. For the collecting system, a large amount of collecting data needs to be processed every day, and the collecting data is large in data magnitude and frequent to update. Therefore, by adopting the method for processing the large-batch data, the method can process the large-batch data rapidly and efficiently by combining the optimistic lock and the pessimistic lock, and ensure that the data cannot be repeatedly grabbed and leaked.
FIG. 1 is a flow chart of a method for processing bulk data according to an embodiment of the present invention. As shown, the method includes the following steps S110-S140.
S110, acquiring all batch data to be processed from a preset database and distributing batch numbers to each batch data to be processed.
In an embodiment, the preset database refers to a database of the collect system, and collect data of the collect system is stored in the preset database. The batch data to be processed refers to the collection data to be processed in batch units, and the batches of the batch data to be processed are divided in advance according to the data size, for example, the data size of the batch data to be processed is 100G in total, and the batch data to be processed is divided into 100 batches by taking 1G as one batch. The lot number refers to the number assigned to the lot Data to be processed, and the Data to be processed of each lot is assigned a lot number, for example, data processing001. Processing the batch data to be processed by the batch number can ensure that processing is not missed.
In one embodiment, as shown in FIG. 2, the step S110 may include steps S111-S113.
S111, acquiring all batch data to be processed from a preset database.
S111, distributing batch numbers according to the batch number of the batch data to be processed.
S112, sorting the batch data to be processed according to the batch numbers from small to large.
In an embodiment, all the batch data to be processed are first obtained from the preset database, then a batch number is allocated to the batch data to be processed of each batch according to the batch number of the obtained batch data to be processed, for example, the batch number of the batch data to be processed is 100, then a batch number is allocated to the 100 batches of batch data to be processed from 0 to 100, and finally the batch numbers are sorted in order from small to large. When the batch data to be processed is processed, processing is performed from the smallest batch number to the largest batch number according to the sequencing order, so that the processing is not omitted.
S120, taking the batch data to be processed with one batch number as target batch data, adding an exclusive lock to the target batch data, and processing the target batch data.
In one embodiment, the exclusive lock is a lock mechanism in a pessimistic lock, which is used to lock data during the entire data processing process, and other processes cannot read and write data. In order to avoid inconsistent data reading and writing caused by reading and writing of data in the process of processing the data by other processes, the batch data to be processed of one batch number is used as target batch data, and an exclusive lock mechanism is adopted to lock the target batch data, so that the other processes are prevented from reading and writing the processed target batch data, and the consistency of data reading and writing is maintained.
In one embodiment, as shown in FIG. 3, the step S120 may include steps S121-S124.
S121, taking the batch data to be processed with one batch number as target batch data.
S122, calling an exclusive lock mechanism of the preset database to add an exclusive lock to the target batch data.
S123, processing the target batch data to which the exclusive lock is added.
S124, adding a processing mark to the processed target batch data.
In an embodiment, the exclusive lock is a lock mechanism of the database itself, so when the exclusive lock is added to the target batch data, the exclusive lock mechanism of a preset database is directly called to lock the target batch data, specifically, the target batch data can be locked by adding a for update to an execution statement for processing the target batch data. And processing the target batch data after locking, wherein the processing of the target batch data comprises adding, deleting and changing data, for example, adding 5 ten thousand yuan to loan in Zhang Sanxin days, repaying 1 ten thousand yuan to four tens of thousands of meters, and recording five tens of meters borrowing for five years, and the target batch data processed in batches today comprises adding Zhang Sanxin records adding 5 ten thousand yuan to loan in a preset database, reducing 1 ten thousand yuan to loan amount of four dates, and deleting the expired five tens of meters borrowing record. And adding a processing mark after processing the target batch data, wherein the processing mark indicates that the batch data to be processed of the batch is processed.
S130, if the target batch data processing is completed, acquiring an initial preset field value in the target batch data and submitting the target batch data to the preset database according to the initial preset field value so as to release the exclusive lock.
In one embodiment, the initial preset field value refers to a preset field value when the target batch data is obtained from the preset database for the first time, and the data to be processed of each batch includes a preset field value. The preset field value may be a version number (version) or a timestamp (timestamp). If the preset field value is a version number, the specific form of the preset field value is a field of a digital type, for example, 1, and each time the version number is increased by one after the data updating is completed, for example, the data with the version number of 1 is updated to be 2; if the preset field value is a timestamp, the specific form is a time type field, for example 2018.11.20,00:00, and the timestamp is updated every time the data update is completed. The optimistic lock is implemented by an initial preset field value, when the target batch data is acquired, other threads default to not process the target batch data, but trigger the optimistic lock when the target batch data is submitted to a preset database after being processed, and whether the target batch data is modified by other threads during the process of acquiring the target batch data and completing the target batch data is judged according to the initial preset field value. If the target batch data is not modified by other threads during the period, submitting the processed target batch data to the preset database so as to release an exclusive lock, wherein the other threads can process the target batch data immediately.
In one embodiment, as shown in FIG. 3, the step S130 may include steps S131-S134.
S131, acquiring an initial preset field value from the target batch data.
S132, acquiring a current preset field value of the target batch data from the preset database.
S133, comparing the initial preset field value with the current preset field value.
S134, if the initial preset field value is the same as the current preset field value, submitting the target batch data to the preset database to release the exclusive lock.
In one embodiment, since the preset field value is an added field value in the target lot data, the initial preset field value is obtained from the target lot data. And then reading the current preset field value in the target batch data of the batch from a preset database. And comparing the initial preset field value with the current preset field value of the target batch data, and if the initial preset field value and the current preset field value are the same, indicating that other threads do not modify the target batch data during the process of obtaining the target batch data and completing the process of the target batch data. Since the exclusive lock mechanism of the preset database is automatically released when the database transaction is submitted, the processed target batch data is submitted to the preset database to be updated, and then the exclusive lock of the target batch data is released, and other processes can immediately perform new processing on the target batch data. If the initial preset field value and the current preset field value of the target batch data are different, the target batch data are modified by other threads during the process of obtaining the target batch data and completing the processing of the target batch data, and at the moment, error information is returned to the collecting system, and the collecting system carries out further processing.
And S140, processing the batch data to be processed of the next batch number as target batch data according to the batch number until all the batch data to be processed of all the batch numbers are processed.
In an embodiment, processing is started from the to-be-processed batch data with the minimum batch number, the to-be-processed batch data with the next batch number is processed after the to-be-processed batch data with the minimum batch number is processed, and the to-be-processed batch data with the next batch number is processed one by one according to the sequence of the allocated batch numbers from small to large. For example, the minimum lot number is 1, and when the processing of the batch data to be processed with the lot number of 1 is completed, the batch data to be processed with the lot number of 2 is processed until all the batch data to be processed with the lot number are completely processed. All the batch data to be processed can be ensured to be processed completely according to the processing sequence from small to large by the batch numbers, so that the condition of missing processing is prevented.
In one embodiment, as shown in FIG. 4, the step S140 may include steps S141-S143.
S141, acquiring the batch data to be processed of the next batch number according to the batch number.
S142, judging whether the processing mark exists in the batch data to be processed.
S143, if the to-be-processed batch data has the processing mark, the to-be-processed batch data of the next batch number is used as target batch data for processing.
In an embodiment, before the processing of the batch data to be processed of the next batch number, the currently processed batch data to be processed needs to be judged, whether a processing mark exists in the currently processed batch data to be processed is judged, if the processing mark exists in the currently processed batch data to be processed, the current batch data to be processed is indicated to be processed, the processing of the batch data to be processed of the next batch is started, that is, the processing of the batch data to be processed of the next batch number is started after returning to the step S120 until all the batch data to be processed are processed. If the processing mark does not exist in the current batch data to be processed, which indicates that the current batch data to be processed is abnormal in the processing process, for example, a certain data is missed to be processed or is processed in error, an error message is returned to the collecting system, and the collecting system traces back the batch data to be processed to further process.
When the method processes a large amount of data to be collected, firstly, the data of the batch to be processed is obtained from the preset database in an optimistic lock mode, and the data of the batch to be processed is not locked first, and other threads can still operate the data of the batch to be processed at the moment. And then firstly processing the batch data to be processed of the minimum batch number according to the sequence of the batch numbers, taking the batch data to be processed of the minimum batch number as target batch data, adding an exclusive lock to the target batch data of the batch number when processing the target batch data, wherein other threads can not operate the target batch data of the batch number at the moment, but can still operate the batch data to be processed of other batch numbers at the moment. Then, after the processing of the target batch data of the batch number is completed, triggering an optimistic lock when the target batch data of the batch number is submitted to a preset database to judge whether the target batch data of the batch number is modified by other threads during the process, if the target batch data of the batch number is not modified by other threads during the process, submitting the processed target batch data of the batch number to the preset database to update, and simultaneously releasing an exclusive lock of the target batch data of the batch number, wherein the target batch data of the batch number can be modified by other threads immediately. Therefore, the method for processing the massive data can be used for rapidly capturing massive data, has little influence on other threads when the data is processed, can rapidly perform new processing, ensures that the data cannot be repeatedly captured and missed, and greatly improves the processing speed and efficiency of the massive data.
The embodiment of the invention discloses a data processing method, which comprises the steps of obtaining all batch data to be processed from a preset database and distributing batch numbers to each batch data to be processed; taking the batch data to be processed of one batch number as target batch data, adding an exclusive lock to the target batch data and processing the target batch data; if the target batch data is processed, acquiring an initial preset field value in the target batch data and submitting the target batch data to the preset database according to the initial preset field value so as to release the exclusive lock; and processing the batch data to be processed of the next batch number as target batch data according to the batch number until all the batch data to be processed of all the batch numbers are processed, so that the data can be ensured not to be repeatedly grabbed and not to be processed, and the processing speed of a large amount of data can be improved.
FIG. 6 is a schematic block diagram of a mass data processing apparatus 200 provided in an embodiment of the present invention. As shown in fig. 6, the present invention also provides a mass data processing apparatus 200 corresponding to the above mass data processing method. The mass data processing apparatus 200, which may be configured in a server, includes means for performing the mass data processing method described above. Specifically, referring to fig. 6, the mass data processing apparatus 200 includes: a distribution unit 210, an exclusive unit 220, a release unit 230, and a processing unit 240.
The distribution unit 210 is configured to obtain all the batch data to be processed from the preset database and distribute a batch number to each batch data to be processed.
In one embodiment, as shown in fig. 7, the distribution unit 210 includes: an acquisition subunit 211, an allocation subunit 212, and a sorting unit 213.
The obtaining subunit 211 is configured to obtain all batch data to be processed from the preset database.
A distribution subunit 212, configured to distribute a lot number according to the lot number of the lot data to be processed.
And a sorting unit 213, configured to sort the batch data to be processed from small to large according to the batch number.
An exclusive unit 220, configured to take the batch data to be processed with one of the batch numbers as target batch data, add an exclusive lock to the target batch data, and process the target batch data.
In one embodiment, as shown in fig. 7, the exclusive unit 220 includes: target unit 221, call unit 222, add unit 223, and mark unit 224.
The target unit 221 is configured to take the batch data to be processed of one of the batch numbers as target batch data.
And a calling unit 222, configured to call an exclusive lock mechanism of the preset database to add an exclusive lock to the target batch data.
An adding unit 223 for processing the target lot data to which the exclusive lock has been added.
A marking unit 224, configured to add a processing mark to the target batch data after processing is completed.
And a releasing unit 230, configured to obtain an initial preset field value in the target lot data and submit the target lot data to the preset database according to the initial preset field value to release the exclusive lock if the target lot data processing is completed.
In one embodiment, as shown in fig. 7, the release unit 230 includes: a first acquisition unit 231, a second acquisition unit 232, a comparison unit 233, and a commit unit 234.
A first obtaining unit 231 configured to obtain an initial preset field value from the target lot data;
a second obtaining unit 232, configured to obtain a current preset field value of the target lot data from the preset database;
A comparing unit 233, configured to compare the initial preset field value with the current preset field value;
And a submitting unit 234, configured to submit the target lot data to the preset database to release the exclusive lock if the initial preset field value is the same as the current preset field value.
And a processing unit 240, configured to process the batch data to be processed of the next batch number as target batch data according to the batch number until all the batch data to be processed of all the batch numbers are processed.
In one embodiment, as shown in fig. 7, the processing unit 240 includes: a third acquisition unit 241, a judgment unit 242, and a loop processing unit 243.
A third obtaining unit 241, configured to obtain the batch data to be processed of the next batch number according to the batch number;
a judging unit 242, configured to judge whether the processing mark exists in the batch data to be processed;
The circulation processing unit 243 is configured to process the batch data to be processed of the next batch number as target batch data if the batch data to be processed has the processing flag.
It should be noted that, as will be clearly understood by those skilled in the art, the specific implementation process of the above-mentioned mass data processing apparatus 200 and each unit may refer to the corresponding description in the foregoing method embodiments, and for convenience and brevity of description, the description is omitted here.
The bulk data processing apparatus described above may be implemented in the form of a computer program which is executable on a computer device as shown in fig. 8.
Referring to fig. 8, fig. 8 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 may be a server, where the server may be a stand-alone server or may be a server cluster formed by a plurality of servers.
With reference to FIG. 8, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a method of bulk data processing.
The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.
The internal memory 504 provides an environment for the execution of a computer program 5032 in the non-volatile storage medium 503, which computer program 5032, when executed by the processor 502, causes the processor 502 to perform a high volume data processing method.
The network interface 505 is used for network communication with other devices. It will be appreciated by those skilled in the art that the architecture shown in fig. 8 is merely a block diagram of some of the architecture relevant to the present inventive arrangements and is not limiting of the computer device 500 to which the present inventive arrangements may be implemented, as a particular computer device 500 may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
Wherein the processor 502 is configured to execute a computer program 5032 stored in a memory to implement the steps of: acquiring all batch data to be processed from a preset database and distributing batch numbers to each batch data to be processed; taking the batch data to be processed of one batch number as target batch data, adding an exclusive lock to the target batch data and processing the target batch data; if the target batch data is processed, acquiring an initial preset field value in the target batch data and submitting the target batch data to the preset database according to the initial preset field value so as to release the exclusive lock; and processing the batch data to be processed of the next batch number as target batch data according to the batch number until all the batch data to be processed of all the batch numbers are processed.
In one embodiment, when implementing the step of acquiring all the batch data to be processed from the preset database and assigning a batch number to each batch data to be processed, the processor 502 specifically implements the following steps: acquiring all batch data to be processed from a preset database; distributing batch numbers according to the batch number of the batch data to be processed; and sequencing the batch data to be processed according to the batch numbers from small to large.
In one embodiment, when the step of using the batch data to be processed with one batch number as target batch data, adding an exclusive lock to the target batch data and processing the target batch data is implemented by the processor 502, the following steps are specifically implemented: taking the batch data to be processed of one batch number as target batch data; invoking an exclusive lock mechanism of the preset database to add an exclusive lock to the target batch data; processing the target batch data to which the exclusive lock has been added; and adding a processing mark to the processed target batch data.
In one embodiment, when the step of obtaining the initial preset field value in the target lot data and submitting the target lot data to the preset database according to the initial preset field value to release the exclusive lock is implemented by the processor 502, the following steps are specifically implemented: acquiring an initial preset field value from the target batch data; acquiring a current preset field value of the target batch data from the preset database; comparing the initial preset field value with the current preset field value; and if the initial preset field value is the same as the current preset field value, submitting the target batch data to the preset database to release the exclusive lock.
In an embodiment, when the processor 502 performs the steps of processing the batch data to be processed of the next batch number as the target batch data according to the batch number until all the batch data to be processed of all the batch numbers are processed, the following steps are specifically implemented: acquiring the batch data to be processed of the next batch number according to the batch number; judging whether the processing mark exists in the batch data to be processed or not; and if the to-be-processed batch data has the processing mark, processing the to-be-processed batch data of the next batch number as target batch data.
It should be appreciated that in embodiments of the present application, the Processor 502 may be a central processing unit (Central Processing Unit, CPU), the Processor 502 may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL processors, DSPs), application SPECIFIC INTEGRATED Circuits (ASICs), off-the-shelf Programmable gate arrays (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. Wherein the general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Those skilled in the art will appreciate that all or part of the flow in a method embodying the above described embodiments may be accomplished by computer programs instructing the relevant hardware. The computer program comprises program instructions, and the computer program can be stored in a storage medium, which is a computer readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.
Accordingly, the present invention also provides a storage medium. The storage medium may be a computer readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. The program instructions, when executed by the processor, cause the processor to perform the steps of: acquiring all batch data to be processed from a preset database and distributing batch numbers to each batch data to be processed; taking the batch data to be processed of one batch number as target batch data, adding an exclusive lock to the target batch data and processing the target batch data; if the target batch data is processed, acquiring an initial preset field value in the target batch data and submitting the target batch data to the preset database according to the initial preset field value so as to release the exclusive lock; and processing the batch data to be processed of the next batch number as target batch data according to the batch number until all the batch data to be processed of all the batch numbers are processed.
In one embodiment, when the processor executes the program instructions to implement the step of obtaining all the batch data to be processed from the preset database and assigning a batch number to each batch data to be processed, the method specifically includes the following steps: acquiring all batch data to be processed from a preset database; distributing batch numbers according to the batch number of the batch data to be processed; and sequencing the batch data to be processed according to the batch numbers from small to large.
In one embodiment, when the processor executes the program instructions to implement the step of using the batch data to be processed of one of the batch numbers as target batch data, adding an exclusive lock to the target batch data and processing the target batch data, the processor specifically implements the following steps: taking the batch data to be processed of one batch number as target batch data; invoking an exclusive lock mechanism of the preset database to add an exclusive lock to the target batch data; processing the target batch data to which the exclusive lock has been added; and adding a processing mark to the processed target batch data.
In one embodiment, when the processor executes the program instructions to obtain an initial preset field value in the batch data to be processed and submit the batch data to be processed to the preset database according to the initial preset field value to release the exclusive lock, the processor specifically implements the following steps: acquiring an initial preset field value from the target batch data; acquiring a current preset field value of the target batch data from the preset database; comparing the initial preset field value with the current preset field value; and if the initial preset field value is the same as the current preset field value, submitting the target batch data to the preset database to release the exclusive lock.
In an embodiment, when the processor executes the program instructions to implement the step of processing the batch data to be processed of the next batch number as target batch data according to the batch number until all the processing of the batch data to be processed of all the batch numbers is completed, the method specifically includes the following steps: acquiring the batch data to be processed of the next batch number according to the batch number; judging whether the processing mark exists in the batch data to be processed or not; and if the to-be-processed batch data has the processing mark, processing the to-be-processed batch data of the next batch number as target batch data.
The storage medium may be a U-disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk, or other various computer-readable storage media that can store program codes.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed.
The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be combined, divided and deleted according to actual needs. In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The integrated unit may be stored in a storage medium if implemented in the form of a software functional unit and sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention is essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a terminal, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention.
While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.
Claims (6)
1. A method of mass data processing, comprising:
Acquiring all batch data to be processed from a preset database in an optimistic lock mode, and distributing batch numbers to each batch data to be processed;
Taking the batch data to be processed of one batch number as target batch data; invoking an exclusive lock mechanism of the preset database to add an exclusive lock to the target batch data; processing the target batch data to which the exclusive lock has been added; adding a processing mark to the processed target batch data;
Triggering an optimistic lock to judge whether the target batch data is processed, and acquiring an initial preset field value from the target batch data if the target batch data is processed; acquiring a current preset field value of the target batch data from the preset database; comparing the initial preset field value with the current preset field value; if the initial preset field value is the same as the current preset field value, submitting the target batch data to the preset database to release the exclusive lock, wherein the preset field value is a preset field value when the target batch data is acquired from the preset database for the first time, the data to be processed of each batch contains one preset field value, the preset field value is a version number or a timestamp, the version number is increased by one every time after the data updating is completed, and the time is updated every time the data updating is completed;
acquiring the batch data to be processed of the next batch number according to the batch number; judging whether the processing mark exists in the batch data to be processed or not; and if the to-be-processed batch data has the processing mark, processing the to-be-processed batch data of the next batch number as target batch data.
2. The method according to claim 1, wherein the steps of acquiring all batch data to be processed from a predetermined database and assigning a batch number to each batch data to be processed, comprise:
Acquiring all batch data to be processed from a preset database;
distributing batch numbers according to the batch number of the batch data to be processed;
And sequencing the batch data to be processed according to the batch numbers from small to large.
3. A mass data processing apparatus, comprising:
the distribution unit is used for acquiring all batch data to be processed from a preset database in an optimistic lock mode and distributing batch numbers to each batch data to be processed;
An exclusive unit for taking the batch data to be processed of one batch number as target batch data; invoking an exclusive lock mechanism of the preset database to add an exclusive lock to the target batch data; processing the target batch data to which the exclusive lock has been added; adding a processing mark to the processed target batch data;
The release unit is used for triggering the optimistic lock to judge to acquire an initial preset field value from the target batch data if the target batch data is processed; acquiring a current preset field value of the target batch data from the preset database; comparing the initial preset field value with the current preset field value; if the initial preset field value is the same as the current preset field value, submitting the target batch data to the preset database to release the exclusive lock, wherein the preset field value is a preset field value when the target batch data is acquired from the preset database for the first time, the data to be processed of each batch contains one preset field value, the preset field value is a version number or a timestamp, the version number is increased by one every time after the data updating is completed, and the time is updated every time the data updating is completed;
The processing unit is used for acquiring the batch data to be processed of the next batch number according to the batch number; judging whether the processing mark exists in the batch data to be processed or not; and if the to-be-processed batch data has the processing mark, processing the to-be-processed batch data of the next batch number as target batch data.
4. A mass data processing device as claimed in claim 3, wherein the allocation unit comprises:
The acquisition subunit is used for acquiring all batch data to be processed from a preset database;
the distribution subunit is used for distributing batch numbers according to the batch number of the batch data to be processed;
And the sequencing unit is used for sequencing the batch data to be processed according to the batch numbers from small to large.
5. A computer device, characterized in that it comprises a memory on which a computer program is stored and a processor which, when executing the computer program, implements the method according to any of claims 1-2.
6. A computer readable storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method according to any of claims 1-2.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910367480.8A CN110263048B (en) | 2019-05-05 | 2019-05-05 | Method and device for processing large-batch data, computer equipment and storage medium |
| PCT/CN2019/117745 WO2020224216A1 (en) | 2019-05-05 | 2019-11-13 | Bulk data processing method and apparatus, and computer device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910367480.8A CN110263048B (en) | 2019-05-05 | 2019-05-05 | Method and device for processing large-batch data, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110263048A CN110263048A (en) | 2019-09-20 |
| CN110263048B true CN110263048B (en) | 2024-05-10 |
Family
ID=67914167
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910367480.8A Active CN110263048B (en) | 2019-05-05 | 2019-05-05 | Method and device for processing large-batch data, computer equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110263048B (en) |
| WO (1) | WO2020224216A1 (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110263048B (en) * | 2019-05-05 | 2024-05-10 | 平安科技（深圳）有限公司 | Method and device for processing large-batch data, computer equipment and storage medium |
| CN111126948A (en) * | 2019-12-02 | 2020-05-08 | 中国建设银行股份有限公司 | Processing method and device for approval process |
| CN111611229B (en) * | 2020-06-11 | 2023-05-26 | 上海乾臻信息科技有限公司 | Data storage method and device for data storage |
| CN112948096A (en) * | 2021-04-07 | 2021-06-11 | 中国工商银行股份有限公司 | Batch scheduling method, device and equipment |
| CN113094185A (en) * | 2021-04-13 | 2021-07-09 | 中国工商银行股份有限公司 | Method and device for detecting parallel data processing abnormity of database batch program |
| CN114756595B (en) * | 2022-06-14 | 2022-08-26 | 希望知舟技术(深圳)有限公司 | Data processing method and related device for database |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6850938B1 (en) * | 2001-02-08 | 2005-02-01 | Cisco Technology, Inc. | Method and apparatus providing optimistic locking of shared computer resources |
| US7734604B1 (en) * | 2000-11-16 | 2010-06-08 | Teradata Us, Inc. | Locking data in a database after an operation has begun |
| CN104199869A (en) * | 2014-08-18 | 2014-12-10 | 中国建设银行股份有限公司 | Service batch processing method, service server and system |
| CN107908344A (en) * | 2017-11-08 | 2018-04-13 | 平安科技（深圳）有限公司 | Method, apparatus, equipment and the computer-readable recording medium for preventing interface from shaking |
| CN107957903A (en) * | 2017-11-13 | 2018-04-24 | 中国平安财产保险股份有限公司 | Asynchronous task scheduling method, server and storage medium |
| CN109460303A (en) * | 2018-09-11 | 2019-03-12 | 阿里巴巴集团控股有限公司 | A kind of data processing method and device, a kind of calculating equipment and storage medium |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015149885A1 (en) * | 2014-04-01 | 2015-10-08 | Huawei Technologies Co.,Ltd | Method for querying and updating entries in a data base |
| CN109344164B (en) * | 2018-10-31 | 2021-03-19 | 迈普通信技术股份有限公司 | Data storage method and device |
| CN109542922B (en) * | 2018-11-29 | 2023-04-07 | 泰康保险集团股份有限公司 | Processing method for real-time service data and related system |
| CN109656980A (en) * | 2018-12-27 | 2019-04-19 | Oppo（重庆）智能科技有限公司 | Data processing method, electronic equipment, device and readable storage medium storing program for executing |
| CN110263048B (en) * | 2019-05-05 | 2024-05-10 | 平安科技（深圳）有限公司 | Method and device for processing large-batch data, computer equipment and storage medium |
-
2019
- 2019-05-05 CN CN201910367480.8A patent/CN110263048B/en active Active
- 2019-11-13 WO PCT/CN2019/117745 patent/WO2020224216A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7734604B1 (en) * | 2000-11-16 | 2010-06-08 | Teradata Us, Inc. | Locking data in a database after an operation has begun |
| US6850938B1 (en) * | 2001-02-08 | 2005-02-01 | Cisco Technology, Inc. | Method and apparatus providing optimistic locking of shared computer resources |
| CN104199869A (en) * | 2014-08-18 | 2014-12-10 | 中国建设银行股份有限公司 | Service batch processing method, service server and system |
| CN107908344A (en) * | 2017-11-08 | 2018-04-13 | 平安科技（深圳）有限公司 | Method, apparatus, equipment and the computer-readable recording medium for preventing interface from shaking |
| CN107957903A (en) * | 2017-11-13 | 2018-04-24 | 中国平安财产保险股份有限公司 | Asynchronous task scheduling method, server and storage medium |
| CN109460303A (en) * | 2018-09-11 | 2019-03-12 | 阿里巴巴集团控股有限公司 | A kind of data processing method and device, a kind of calculating equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2020224216A1 (en) | 2020-11-12 |
| CN110263048A (en) | 2019-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110263048B (en) | Method and device for processing large-batch data, computer equipment and storage medium | |
| US8671085B2 (en) | Consistent database recovery across constituent segments | |
| US11657035B2 (en) | Methods, devices and systems for real-time checking of data consistency in a distributed heterogenous storage system | |
| US9251195B2 (en) | Method of managing database | |
| US20160179865A1 (en) | Method and system for concurrency control in log-structured merge data stores | |
| US8788474B2 (en) | Inode event notification for cluster file systems | |
| US20110023044A1 (en) | Scheduling highly parallel jobs having global interdependencies | |
| CN110581887A (en) | Data processing method, device, block chain node and storage medium | |
| US7752399B2 (en) | Exclusion control method and information processing apparatus | |
| CN113256355B (en) | Method, device, medium, equipment and system for determining integral rights and interests in real time | |
| US20240097902A1 (en) | Parallel block processing in blockchains | |
| US20130198138A1 (en) | Model for capturing audit trail data with reduced probability of loss of critical data | |
| CN116701415A (en) | Execution method of cross-chain transaction and related equipment | |
| CN116226081A (en) | Database elastic expansion method and device, electronic equipment and storage medium | |
| CN117291741A (en) | Account data processing method, device, equipment, storage medium and program product | |
| Tong et al. | TELL: Efficient Transaction Execution Protocol Towards Leaderless Consensus | |
| CN114116614A (en) | Log storage method and device, computer equipment and storage medium | |
| CN117785900A (en) | Data updating method, device, computer equipment and storage medium | |
| CN117593125A (en) | Service data processing method, device, computer equipment and storage medium | |
| CN116643862A (en) | Data processing method, distributed database management system and equipment | |
| CN116414772A (en) | Data dump method, device, equipment and storage medium | |
| CN115439226A (en) | Transaction processing method and device | |
| CN117806799A (en) | Batch task distributed scheduling method and device | |
| CN118689618A (en) | Task processing method, device, apparatus, readable storage medium and program product | |
| CN116258479A (en) | Problem positioning method and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |