CN111352948A - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111352948A CN111352948A CN202010242018.8A CN202010242018A CN111352948A CN 111352948 A CN111352948 A CN 111352948A CN 202010242018 A CN202010242018 A CN 202010242018A CN 111352948 A CN111352948 A CN 111352948A
- Authority
- CN
- China
- Prior art keywords
- fragment
- target
- target data
- data
- stationing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 23
- 239000012634 fragment Substances 0.000 claims abstract description 273
- 238000000034 method Methods 0.000 claims abstract description 66
- 238000012545 processing Methods 0.000 claims abstract description 54
- 230000008569 process Effects 0.000 claims abstract description 50
- 238000013467 fragmentation Methods 0.000 claims description 8
- 238000006062 fragmentation reaction Methods 0.000 claims description 8
- 238000013507 mapping Methods 0.000 claims description 7
- 238000012217 deletion Methods 0.000 claims description 3
- 230000037430 deletion Effects 0.000 claims description 3
- 238000004590 computer program Methods 0.000 claims description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: the control target stationing thread acquires the fragment number of the target data fragment to be processed from the fragment control table, and acquires a plurality of target data with the corresponding fragment number being the same as the fragment number of the target data fragment from the quick access table according to the fragment number of the target data fragment, wherein the target data fragment comprises a plurality of target data, the plurality of target data are data corresponding to the same hot spot field in the database, and the control target stationing thread sequentially processes the plurality of target data. It has the following technical effects: the target stationing thread processes a plurality of target data in sequence in a sequential execution mode, the complexity of the data processing process is low, the performance damage degree is low, and meanwhile, the hot spot field can be updated according to expectation.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
During the operation of the database, data in some fields need to be updated frequently and concurrently, and these fields may be referred to as hot-spot fields. If a plurality of received data are updated to a certain hot spot field at the same time, how to update the hot spot field is very important.
At present, when a plurality of data corresponding to the same hot spot field are received at the same time, the hot spot field is updated in parallel according to the data.
However, the above-mentioned update process may result in a high complexity and a high performance loss of the update process on one hand, and the parallel update may result in that the hot spot field cannot be updated as expected on the other hand.
Disclosure of Invention
The invention provides a data processing method, a data processing device, data processing equipment and a data processing storage medium, which are used for solving the technical problems that when a hot spot field is updated at present, the complexity is high, the performance loss is high, and the updating cannot be performed according to expectation.
In a first aspect, an embodiment of the present invention provides a data processing method, including:
the control target stationing thread acquires the fragment number of the target data fragment to be processed from the fragment control table;
controlling the target stationing thread to acquire a plurality of target data with corresponding fragment numbers identical to the fragment numbers of the target data fragments from a quick access table according to the fragment numbers of the target data fragments; the target data fragments comprise a plurality of target data, and the target data are data corresponding to the same hot spot fields in the database;
and controlling the target stationing thread to process a plurality of target data in sequence.
In a second aspect, an embodiment of the present invention provides a data processing apparatus, including:
the first acquisition module is used for controlling the target stationing thread to acquire the fragment number of the target data fragment to be processed from the fragment control table;
the second acquisition module is used for controlling the target stationing thread to acquire a plurality of target data with corresponding fragment numbers identical to the fragment numbers of the target data fragments from a quick access table according to the fragment numbers of the target data fragments; the target data fragments comprise a plurality of target data, and the target data are data corresponding to the same hot spot fields in the database;
and the processing module is used for controlling the target stationing thread to sequentially process a plurality of target data.
In a third aspect, an embodiment of the present invention further provides a computer device, where the computer device includes:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the data processing method as provided in the first aspect.
In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the data processing method as provided in the first aspect.
The embodiment provides a data processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: the control target stationing thread acquires the fragment number of the target data fragment to be processed from the fragment control table, and acquires a plurality of target data with the corresponding fragment number being the same as the fragment number of the target data fragment from the quick access table according to the fragment number of the target data fragment, wherein the target data fragment comprises a plurality of target data, the plurality of target data are data corresponding to the same hot spot field in the database, and the control target stationing thread sequentially processes the plurality of target data. It has the following technical effects: compared with the situation that the updating process is high in complexity and high in performance loss and the hot field cannot be updated according to expectation due to the fact that the target stationing thread processes the plurality of target data in sequence in a parallel mode, the data processing method provided by the embodiment avoids the situation that the plurality of target data are processed in a parallel mode, the data processing process is low in complexity and low in performance damage degree, and meanwhile the hot field can be updated according to the expectation.
Drawings
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating table interactions according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention. The embodiment is suitable for the scene of updating the hot spot fields in the database. The present embodiment may be performed by a data processing apparatus, which may be implemented by means of software and/or hardware, which may be integrated in a computer device. As shown in fig. 1, the data processing method provided in this embodiment includes the following steps:
step 101: and the control target stationing thread acquires the fragment number of the target data fragment to be processed from the fragment control table.
Specifically, in the present embodiment, a plurality of stationing threads are provided, and the number of stationing threads may be specified in the yml configuration file.
The fragment control table in this embodiment stores the fragment number of each data fragment to distinguish different data fragments. That is, the primary key of the fragmentation control table is the fragmentation number. A data shard in this embodiment refers to a collection of data used to update the same hotspot field in a database. The hot spot field in this embodiment may be a field predefined according to a service scenario and an actual requirement. The hot spot field may be one or more fields, such as "status", "progress", "timestamp of job", "error code", and "error description" fields.
And the plurality of stationed threads acquire the fragment numbers of the target data fragments to be processed from the fragment control table in a competitive mode. In this embodiment, for convenience of description, a camping thread that competes for a data fragment is referred to as a target camping thread, and a data fragment that competes for a target data fragment is referred to as a target data fragment.
The target data segment in the present embodiment includes a plurality of target data. That is, the target data slice in this embodiment is substantially a collection of a plurality of target data. The target data are data corresponding to the same hot spot field in the database, that is, the target data are data for updating the same hot spot field in the database.
The following describes in detail how the target camping thread obtains the fragment number of the target data fragment.
In one implementation, the fragmentation control table further includes a lock flag. Wherein, a fragment number corresponds to a lock mark. The lock flag is used to identify whether the data slice is currently processed. When the lock mark is in a locked state, the data fragment is currently processed; when the lock flag is in the released state, it indicates that the data fragment is not currently processed.
In the implementation mode, the control target stationing thread acquires a lock mark of a first data fragment from a fragment control table; and if the lock mark of the first data fragment is in a release state, controlling the target stationing thread to determine the first data fragment as the target data fragment, and determining the fragment number of the first data fragment as the fragment number of the target data fragment. The first data slice in this embodiment may refer to any data slice in the slice control table. In the above process, the control target gatekeeper thread acquires the lock flag of any data fragment (referred to as a first data fragment for convenience of description) from the fragment control table, and if the lock flag is in a release state, it indicates that the first data fragment is not currently processed. And controlling the stationed thread to take the first data fragment as a target data fragment and take the fragment code of the first data fragment as the fragment code of the target data fragment.
Further, based on the foregoing implementation, the fragmentation control table further includes: token and execution start time. One slice number corresponds to one lock flag, one token, and one execution start time. After the control target stationing thread processes the target data in sequence, the control target stationing thread modifies the lock mark of the target data fragment in the fragment control table into a lock state to indicate that the target data fragment is currently processed; controlling the target stationing thread to update the token of the target data fragment in the fragment control table to the token of the target stationing thread so as to indicate that the target data fragment is currently processed by the target stationing thread; after the target stationing thread starts to process the target data in sequence, the control target stationing updates the execution start time of the target data fragments in the fragment control table to indicate when the target data fragments are processed from the beginning. There is no timing relationship among the steps of modifying the lock flag, updating the token, and updating the execution start time.
In another implementation manner, in the process of acquiring the fragment number of the target data fragment, if the lock flag of the first data fragment is in a locked state, determining whether the absolute value of the difference value between the execution start time of the first data fragment and the current time is greater than a preset time threshold; if the absolute value of the difference value between the execution starting time of the first data fragment and the current time is larger than the preset time threshold value, determining the first data fragment as a target data fragment, and determining the fragment number of the first data fragment as the fragment number of the target data fragment; the control target stationing thread updates the token of the target data fragment in the fragment control table into the token of the target stationing thread; and after the target stationing thread starts to process the target data in sequence, controlling the target stationing thread to update the execution starting time of the target data fragments in the fragment control table.
In this implementation, the lock flag of the first data fragment is in a locked state, and the absolute value of the difference between the execution start time of the first data fragment and the current time is greater than the preset time threshold, which indicates that although the first data fragment is currently processed, the processing time has already expired, that is, the processing process is expired due to an abnormal condition such as a failure occurring in the thread on the premises that previously processed the first data fragment. In order to enable the target data fragment to be continuously executed, the target stationing thread determines the first data fragment as the target data fragment, determines the fragment number of the first data fragment as the fragment number of the target data fragment, and updates the token of the target data fragment in the fragment control table to the token of the target stationing thread; and after the target stationing thread starts to process the target data in sequence, controlling the target stationing thread to update the execution starting time of the target data fragments in the fragment control table.
Further, the present embodiment may further include the following steps: controlling the target stationing thread to judge whether the token of the target data fragment is the token of the target stationing thread in the process of sequentially processing the target data; and if the token of the target data fragment is not the token of the target stationing thread, controlling the target stationing thread to stop processing the target data. In the process of sequentially processing the target data fragments, the target camping thread may determine whether the token of the target data fragment is the token of the target camping thread at a preset time interval. And if the token of the target data fragment is determined not to be the token of the target stationing thread, the target data fragment is processed by other stationing threads, and the target stationing thread is controlled to stop processing the target data. So as to avoid the target data fragment being processed repeatedly and save processing resources.
In another implementation manner, in the process of acquiring the fragment number of the target data fragment, if the lock flag of the first data fragment is in a locked state and the absolute value of the difference between the execution start time of the first data fragment and the current time is less than or equal to the preset time threshold, another data fragment different from the first data fragment in the fragment control table is used as a new first data fragment, and the process of acquiring the fragment number of the target data fragment is repeated until the fragment number of the target data fragment is determined.
Step 102: and controlling the target stationing thread to acquire a plurality of target data with the corresponding fragment numbers identical to the fragment numbers of the target data fragments from the quick access table according to the fragment numbers of the target data fragments.
The target data fragments comprise a plurality of target data, and the target data are data corresponding to the same hot spot fields in the database.
Specifically, the quick access table in the present embodiment stores unprocessed data. Further, the quick access table may further store a fragment number, a database clock, a module deployment name, and a local sequence number corresponding to unprocessed data. The primary key of the quick access table may be: a tile number, a database clock, a module deployment name, and a local sequence number. The target data in this embodiment is a part of unprocessed data.
The following is a detailed description of how the target data is stored in the quick access table. The target data in this embodiment is transmitted from the transmitting device to the data processing apparatus. The data processing device receives a plurality of target data sent by the sending equipment, determines the fragment number of the target data according to the mapping relation between the data identifier and the fragment number and the identifier of the target data, and stores the target data in the position corresponding to the fragment number of the target data in the quick access table. The sending device in this embodiment may be a device that generates target data. In this embodiment, the slice numbers of the plurality of target data are the same.
It should be noted that, when the target data is stored in the location corresponding to the fragment number of the target data in the quick access table, since the plurality of target data correspond to one fragment number, each target data may be stored in the corresponding location in combination with the database clock, the module deployment name, and the local sequence number.
The identifier of the target data in this embodiment may be a hot key field of the target data or a job number.
In one implementation manner, a mapping relationship between a hot spot key field or a job number of data and a fragment number is configured in advance, after a target data is received, a fragment number corresponding to the target data is determined from the mapping relationship according to the hot spot key field or the job number of the target data, and then the target data is stored in a position corresponding to the fragment number in a quick access table.
In another implementation manner, a mapping relationship between a value obtained by adding the hash value of the hot key field of the data or the job number and the fragment number is configured in advance, after receiving a target data, a value obtained by adding the hash value of the hot key field of the target data or the job number is determined, the fragment number corresponding to the target data is determined from the mapping relationship, and then the target data is stored in a position corresponding to the fragment number in the quick access table. Specifically, a hash value may be calculated by using a hash algorithm built in a Java standard version Development kit (JDK), a remainder value of the hash value is determined, and a fragment number corresponding to the target data is determined according to the remainder value.
The number of bits of the segment number can be set according to actual requirements. Alternatively, the slice number may be 4 bits.
In this embodiment, the transmitting device may transmit the target data through a kaffa (kafka) middleware. The sending device may send data (which includes target data) in batches to the data processing apparatus through the kafka middleware. Correspondingly, the data processing apparatus in the present embodiment receives a plurality of target data transmitted by the transmitting device through the kafka middleware. At the time of specific reception, the data processing apparatus locates specific data in the kafka middleware according to the current data subscript in the kafka middleware, and acquires the data.
Furthermore, a flow meter is also provided in the present embodiment. All data sent to the data processing device by the sending equipment are recorded in the flow meter, so that subsequent checking is facilitated, and data processing omission is avoided.
After receiving the plurality of target data, the data processing device stores the plurality of target data in the flow meter by adopting an adding operation so as to compare the data in the flow meter with the processed data. Wherein the adding operation is an operation of adding according to the main key bar of the flow meter. In this embodiment, the increase operation of the flow meter is a micro operation that is performed according to the single main key of the flow meter, and compared with a batch operation mode, a large amount of concurrent operation databases can be avoided. The conventional operation of adding a plurality of pieces of data can be carried out, the plurality of pieces of data can be operated, and the archiving log of the database can be rapidly increased due to the nesting of the plurality of pieces of data, and even the database can not be accessed after the disk space is full.
After storing the target data in the flow meter, the data processing apparatus modifies the current data index in the kafka middleware to the index of the next batch of yet unsent data in the kafka middleware, committing the kafka transaction.
In this embodiment, after the control target gatekeeper thread acquires the segment numbers of the target data segments, a plurality of target data whose corresponding segment numbers are the same as the segment numbers of the target data segments are acquired from the quick access table.
Step 103: and controlling the target stationing thread to sequentially process a plurality of target data.
Specifically, after the target camping thread acquires the plurality of target data, the target camping thread may sequentially analyze the plurality of target data according to a sequence defined by the target camping thread.
In one implementation, the quick access table further includes a generation time stamp of the target data. Based on this implementation, one possible data processing procedure is as follows.
And controlling the target stationing thread to analyze the first target data and acquiring a generation timestamp of the target data. And comparing the timestamp of the current value in the hot spot field corresponding to the target data in the database. And if the timestamp of the current value in the hot spot field corresponding to the target data is earlier than the generation timestamp of the target data, updating the value of the hot spot field corresponding to the target data into the target data. And if the current timestamp in the hot spot field corresponding to the target data is later than the generation timestamp of the target data, indicating that the target data is expired data, and discarding the target data.
And after the first target data is processed, controlling the target stationing thread to process the second target data according to the implementation mode until all the target data are processed, namely finishing the processing of the target data fragments. It should be noted that the order in which the target data is processed by the target standing-watch thread may be predefined.
Currently, processing multiple data corresponding to the same hotspot field in a parallel manner leads to higher complexity and higher performance loss of the updating process, and leads to the situation that the hotspot field cannot be updated as expected, for example, the concurrent execution of an SQLupdate statement leads to unknown updating results. In this embodiment, the target stationing thread processes the plurality of target data in a sequential execution manner, so that the plurality of target data are prevented from being processed in a parallel manner, the complexity of the data processing process is low, the performance damage is low, and meanwhile, the hot spot field corresponding to the target data can be updated as expected.
And after the control target stationing thread processes the plurality of target data in sequence, deleting the plurality of target data from the quick access table by adopting a deleting operation. The deleting operation is an operation of deleting according to the main key single bar in the quick access table.
Deleting the processed target data can enable the quick access table to maintain better performance, so as to realize quick writing and quick reading, and improve the data processing speed of the embodiment.
The deletion operation in this embodiment is a "micro" operation that deletes the data according to the single key in the quick access table, and compared with a batch operation mode, a large number of concurrent operations on the database can be avoided. The conventional operation of deleting a plurality of pieces of data can be carried out, the plurality of pieces of data can be operated, and the archiving log of the database can be rapidly increased due to the nesting of the plurality of pieces of data, and even the database can not be accessed after the disk space is full.
Further, in order to facilitate monitoring of the processing process of the target stationing thread, in this embodiment, the target stationing thread is controlled to send the target data currently being processed to the current state table in the process of processing the target data. The current status table may be displayed on a screen to facilitate monitoring of the data processing process.
To sum up, the data processing process of the present embodiment involves a plurality of tables: the system comprises a flow meter, a quick access meter, a fragment control meter and a current state meter. FIG. 2 is a diagram illustrating interactions between tables according to an embodiment of the present invention. As shown in fig. 2, all data transmitted from the transmission device to the data processing apparatus in the present embodiment is stored in the flow meter for subsequent checking. The data stored in the quick access table is unprocessed, and after the target data is processed by the target stationing thread, the processed target data is deleted from the quick access table. And after the target stationing thread acquires the fragment number of the target data fragment from the fragment control table, acquiring a plurality of target data with the corresponding fragment number same as that of the target data fragment from the quick access table. In the process that the control target stationing thread processes a plurality of target data in sequence, the target data currently being processed may be displayed in the current state table.
The data processing method provided by the embodiment comprises the following steps: the control target stationing thread acquires the fragment number of the target data fragment to be processed from the fragment control table, and acquires a plurality of target data with the corresponding fragment number being the same as the fragment number of the target data fragment from the quick access table according to the fragment number of the target data fragment, wherein the target data fragment comprises a plurality of target data, the plurality of target data are data corresponding to the same hot spot field in the database, and the control target stationing thread sequentially processes the plurality of target data. It has the following technical effects: compared with the situation that the updating process is high in complexity and high in performance loss and the hot field cannot be updated according to expectation due to the fact that the target stationing thread processes the plurality of target data in sequence in a parallel mode, the data processing method provided by the embodiment avoids the situation that the plurality of target data are processed in a parallel mode, the data processing process is low in complexity and low in performance damage degree, and meanwhile the hot field can be updated according to the expectation.
Fig. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention. As shown in fig. 3, the data processing apparatus provided in the present embodiment includes: a first acquisition module 31, a second acquisition module 32 and a processing module 33.
The first obtaining module 31 is configured to control the target camping thread to obtain a fragment number of a target data fragment to be processed from the fragment control table.
And the second obtaining module 32 is configured to control the target gatekeeper thread to obtain, according to the fragment number of the target data fragment, a plurality of target data having the same corresponding fragment number as the fragment number of the target data fragment from the quick access table.
The target data fragments comprise a plurality of target data, and the target data are data corresponding to the same hot spot fields in the database.
And the processing module 33 is configured to control the target camping thread to sequentially process a plurality of target data.
In one implementation, the fragmentation control table further includes: and (6) locking a mark. The first obtaining module 31 is specifically configured to: a control target stationing thread acquires a lock mark of a first data fragment from a fragment control table; and if the lock mark of the first data fragment is in a release state, controlling the target stationing thread to determine the first data fragment as the target data fragment, and determining the fragment number of the first data fragment as the fragment number of the target data fragment.
In another implementation, the fragmentation control table further includes: token and execution start time. The device also includes: the system comprises a modification module, a first updating module and a second updating module.
And the modification module is used for controlling the target stationing thread to modify the lock mark of the target data fragment in the fragment control table into a locking state.
And the first updating module is used for controlling the target stationing thread to update the token of the target data fragment in the fragment control table into the token of the target stationing thread.
And the second updating module is used for controlling the target stationing and updating the execution starting time of the target data fragments in the fragment control table after the target stationing thread starts to process the target data in sequence.
In another implementation, the apparatus further includes: the device comprises a first determining module, a second determining module, a third updating module and a fourth updating module.
The first determining module is configured to determine whether an absolute value of a difference between execution start time of the first data slice and current time is greater than a preset time threshold if the lock flag of the first data slice is in a locked state.
And the second determining module is used for determining the first data fragment as the target data fragment and determining the fragment number of the first data fragment as the fragment number of the target data fragment if the absolute value of the difference value between the execution starting time of the first data fragment and the current time is greater than the preset time threshold.
And the third updating module is used for controlling the target stationing thread to update the token of the target data fragment in the fragment control table into the token of the target stationing thread.
And the fourth updating module is used for controlling the target stationing and updating the execution starting time of the target data fragments in the fragment control table after the target stationing thread starts to process the target data in sequence.
In yet another implementation, the apparatus further includes: the device comprises a judging module and a stopping processing module.
And the judging module is used for controlling the target stationing thread to judge whether the token of the target data fragment is the token of the target stationing thread in the process of sequentially processing the target data.
And the stop processing module is used for controlling the target stationing thread to stop processing the target data if the token of the target data fragment is not the token of the target stationing thread.
Optionally, the apparatus further comprises a deletion module.
And the deleting module is used for deleting the plurality of target data from the quick access table by adopting deleting operation after the target data are processed and completed by the control target stationing thread in sequence. The deleting operation is an operation of deleting according to the main key single bar in the quick access table.
Optionally, the apparatus further comprises: the device comprises a receiving module, a third determining module and a storing module.
And the receiving module is used for receiving a plurality of target data sent by the sending equipment.
And the third determining module is used for determining the fragment number of the target data according to the mapping relation between the data identifier and the fragment number and the identifier of the target data.
And the storage module is used for storing the target data in a position corresponding to the fragment number of the target data in the quick access table.
More specifically, the receiving module is specifically configured to: receiving a plurality of target data transmitted by a transmitting device through kafka middleware; and storing a plurality of target data in the flow meter by adopting an adding operation so as to compare the data in the flow meter with the processed data. Wherein the adding operation is an operation of adding according to the main key bar of the flow meter.
The data processing device provided by the embodiment of the invention can execute the data processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
Fig. 4 is a schematic structural diagram of a computer device according to an embodiment of the present invention. As shown in fig. 4, the computer device includes a processor 40 and a memory 41. The number of the processors 40 in the computer device may be one or more, and one processor 40 is taken as an example in fig. 4; the processor 40 and the memory 41 of the computer device may be connected by a bus or other means, as exemplified by the bus connection in fig. 4.
The memory 41, which is a computer-readable storage medium, may be used for storing software programs, computer-executable programs, and modules, such as program instructions and modules corresponding to the data processing method in the embodiment of the present invention (for example, the first obtaining module 31, the second obtaining module 32, and the processing module 33 in the data processing apparatus). The processor 40 executes various functional applications of the computer device and data processing by executing software programs, instructions, and modules stored in the memory 41, that is, implements the data processing method described above.
The memory 41 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the computer device, and the like. Further, the memory 41 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 41 may further include memory located remotely from processor 40, which may be connected to a computer device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The present invention also provides a storage medium containing computer-executable instructions which, when executed by a computer processor, are operable to perform a method of data processing, the method comprising:
the control target stationing thread acquires the fragment number of the target data fragment to be processed from the fragment control table;
controlling the target stationing thread to acquire a plurality of target data with corresponding fragment numbers identical to the fragment numbers of the target data fragments from a quick access table according to the fragment numbers of the target data fragments; the target data fragments comprise a plurality of target data, and the target data are data corresponding to the same hot spot fields in the database;
and controlling the target stationing thread to process a plurality of target data in sequence.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the operations of the method described above, and may also perform related operations in the data processing method provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions to enable a computer device (which may be a personal computer, a computer device, or a network device) to execute the data processing method according to the embodiments of the present invention.
It should be noted that, in the embodiment of the data processing apparatus, the included units and modules are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. A data processing method, comprising:
the control target stationing thread acquires the fragment number of the target data fragment to be processed from the fragment control table;
controlling the target stationing thread to acquire a plurality of target data with corresponding fragment numbers identical to the fragment numbers of the target data fragments from a quick access table according to the fragment numbers of the target data fragments; the target data fragments comprise a plurality of target data, and the target data are data corresponding to the same hot spot fields in the database;
and controlling the target stationing thread to process a plurality of target data in sequence.
2. The method of claim 1, wherein the fragmentation control table further comprises: a lock flag;
the control target stationing thread acquires the fragment number of the target data fragment to be processed from the fragment control table, and the method comprises the following steps:
controlling the target stationing thread to acquire a lock mark of a first data fragment from the fragment control table;
and if the lock mark of the first data fragment is in a release state, controlling the target stationing thread to determine the first data fragment as the target data fragment, and determining the fragment number of the first data fragment as the fragment number of the target data fragment.
3. The method of claim 2, wherein the fragmentation control table further comprises: token and execution start time;
after the controlling the target stationing thread to process the target data in sequence, the method further includes:
controlling the target stationing thread to modify the lock mark of the target data fragment in the fragment control table into a lock state;
controlling the target stationing thread to update the token of the target data fragment in the fragment control table to the token of the target stationing thread;
and after the target stationing thread starts to process the target data in sequence, controlling the target stationing thread to update the execution starting time of the target data fragments in the fragment control table.
4. The method of claim 3, further comprising:
if the lock flag of the first data fragment is in a lock state, determining whether the absolute value of the difference value between the execution starting time of the first data fragment and the current time is greater than a preset time threshold value;
if the absolute value of the difference value between the execution starting time of the first data fragment and the current time is determined to be larger than the preset time threshold, determining the first data fragment as the target data fragment, and determining the fragment number of the first data fragment as the fragment number of the target data fragment;
controlling the target stationing thread to update the token of the target data fragment in the fragment control table to the token of the target stationing thread;
after the target stationing thread starts to process the target data in sequence, controlling the target stationing thread to update the execution start time of the target data fragments in the fragment control table;
controlling the target stationing thread to judge whether the token of the target data fragment is the token of the target stationing thread in the process of sequentially processing the target data;
and if the token of the target data fragment is not the token of the target stationing thread, controlling the target stationing thread to stop processing the target data.
5. The method according to any one of claims 1-4, further comprising:
after the target stationing thread is controlled to sequentially process and finish the target data, deleting the plurality of target data from the quick access table by adopting deletion operation; and the deleting operation is an operation of deleting according to a main key single bar in the quick access table.
6. The method according to any one of claims 1 to 4, wherein before the controlling target stationing thread obtains the fragment number of the target data fragment to be processed from the fragment control table, the method further comprises:
receiving a plurality of target data sent by a sending device;
determining the fragment number of the target data according to the mapping relation between the data identifier and the fragment number and the identifier of the target data;
and storing the target data in a position corresponding to the fragment number of the target data in the quick access table.
7. The method of claim 6, wherein the receiving the plurality of target data transmitted by the transmitting device comprises:
receiving a plurality of target data sent by the sending equipment through the kafka middleware;
storing a plurality of target data in a flow meter by adopting an adding operation so as to compare the data in the flow meter with the processed data; wherein the adding operation is an adding operation according to a main key single bar of the flow water meter.
8. A data processing apparatus, comprising:
the first acquisition module is used for controlling the target stationing thread to acquire the fragment number of the target data fragment to be processed from the fragment control table;
the second acquisition module is used for controlling the target stationing thread to acquire a plurality of target data with corresponding fragment numbers identical to the fragment numbers of the target data fragments from a quick access table according to the fragment numbers of the target data fragments; the target data fragments comprise a plurality of target data, and the target data are data corresponding to the same hot spot fields in the database;
and the processing module is used for controlling the target stationing thread to sequentially process a plurality of target data.
9. A computer device, characterized in that the computer device comprises:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a data processing method as claimed in any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the data processing method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010242018.8A CN111352948B (en) | 2020-03-31 | 2020-03-31 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010242018.8A CN111352948B (en) | 2020-03-31 | 2020-03-31 | Data processing method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111352948A true CN111352948A (en) | 2020-06-30 |
| CN111352948B CN111352948B (en) | 2023-12-26 |
Family
ID=71193089
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010242018.8A Active CN111352948B (en) | 2020-03-31 | 2020-03-31 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111352948B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130166556A1 (en) * | 2011-12-23 | 2013-06-27 | Daniel Baeumges | Independent Table Nodes In Parallelized Database Environments |
| CN108038239A (en) * | 2017-12-27 | 2018-05-15 | 中科鼎富（北京）科技发展有限公司 | A kind of heterogeneous data source method of standardization management, device and server |
| CN108874552A (en) * | 2018-06-28 | 2018-11-23 | 杭州云英网络科技有限公司 | Distributed lock executes method, apparatus and system, application server and storage medium |
| CN109002484A (en) * | 2018-06-25 | 2018-12-14 | 北京明朝万达科技股份有限公司 | A kind of method and system for sequence consumption data |
| CN109144731A (en) * | 2018-08-31 | 2019-01-04 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer equipment and storage medium |
| CN109491928A (en) * | 2018-11-05 | 2019-03-19 | 深圳乐信软件技术有限公司 | Buffer control method, device, terminal and storage medium |
| CN109976888A (en) * | 2019-03-25 | 2019-07-05 | 苏州达家迎信息技术有限公司 | A kind of data scanning method, device, equipment and storage medium |
-
2020
- 2020-03-31 CN CN202010242018.8A patent/CN111352948B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130166556A1 (en) * | 2011-12-23 | 2013-06-27 | Daniel Baeumges | Independent Table Nodes In Parallelized Database Environments |
| CN108038239A (en) * | 2017-12-27 | 2018-05-15 | 中科鼎富（北京）科技发展有限公司 | A kind of heterogeneous data source method of standardization management, device and server |
| CN109002484A (en) * | 2018-06-25 | 2018-12-14 | 北京明朝万达科技股份有限公司 | A kind of method and system for sequence consumption data |
| CN108874552A (en) * | 2018-06-28 | 2018-11-23 | 杭州云英网络科技有限公司 | Distributed lock executes method, apparatus and system, application server and storage medium |
| CN109144731A (en) * | 2018-08-31 | 2019-01-04 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer equipment and storage medium |
| CN109491928A (en) * | 2018-11-05 | 2019-03-19 | 深圳乐信软件技术有限公司 | Buffer control method, device, terminal and storage medium |
| CN109976888A (en) * | 2019-03-25 | 2019-07-05 | 苏州达家迎信息技术有限公司 | A kind of data scanning method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111352948B (en) | 2023-12-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110737658B (en) | Data fragment storage method, device, terminal and readable storage medium | |
| CN109918187B (en) | Task scheduling method, device, equipment and storage medium | |
| CN109002305B (en) | Method and system for updating equipment program | |
| CN110688305B (en) | Test environment synchronization method, device, medium and electronic equipment | |
| WO2022017052A1 (en) | Method and apparatus for shared device instruction control and device | |
| WO2022126918A1 (en) | Method and apparatus for automatically synchronizing associated scripts, and computer device and storage medium | |
| CN105653556B (en) | Data aging method and device | |
| CN111352948A (en) | Data processing method, device, equipment and storage medium | |
| US20160157093A1 (en) | System and method for updating firmware through hetero-network | |
| CN111367869A (en) | Mirror image file processing method and device, storage medium and electronic equipment | |
| CN115499487B (en) | Updating method and device of server configuration file, storage medium and equipment | |
| CN111385613B (en) | Television system repairing method, storage medium and application server | |
| CN113284039B (en) | Bitmap management method, device and equipment and readable storage medium | |
| CN112463786B (en) | Data synchronization method, system, server and storage medium | |
| CN115328987A (en) | Database switching method and device, computer equipment and storage medium | |
| CN107872493B (en) | Information processing method, terminal and server | |
| CN111147400B (en) | Method and device for command line configuration synchronization and electronic equipment | |
| CN115809150A (en) | Distributed deadlock detection method and device and electronic equipment | |
| CN111258808B (en) | Method, apparatus, device and medium for data automation management | |
| CN113553488A (en) | Method and device for updating index data in search engine, electronic equipment and medium | |
| CN111382016A (en) | Method and device for connecting test terminal and computer | |
| CN109542631A (en) | A kind of recurrence method, apparatus, server and the storage medium of standby host | |
| CN117407369B (en) | Operation log recording method and device and electronic equipment | |
| CN113238862B (en) | Distributed task scheduling method and device | |
| CN114860157A (en) | Data loading method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220914 Address after: 25 Financial Street, Xicheng District, Beijing 100033 Applicant after: CHINA CONSTRUCTION BANK Corp. Address before: 25 Financial Street, Xicheng District, Beijing 100033 Applicant before: CHINA CONSTRUCTION BANK Corp. Applicant before: Jianxin Financial Science and Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |