CN111352948B - Data processing method, device, equipment and storage medium - Google Patents
Data processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN111352948B CN111352948B CN202010242018.8A CN202010242018A CN111352948B CN 111352948 B CN111352948 B CN 111352948B CN 202010242018 A CN202010242018 A CN 202010242018A CN 111352948 B CN111352948 B CN 111352948B
- Authority
- CN
- China
- Prior art keywords
- target data
- target
- fragment
- data
- resident thread
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 23
- 239000012634 fragment Substances 0.000 claims abstract description 230
- 238000000034 method Methods 0.000 claims abstract description 66
- 238000012545 processing Methods 0.000 claims abstract description 57
- 230000008569 process Effects 0.000 claims abstract description 46
- 238000013507 mapping Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 2
- 230000000694 effects Effects 0.000 abstract description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 238000013467 fragmentation Methods 0.000 description 4
- 238000006062 fragmentation reaction Methods 0.000 description 4
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a data processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: the control target resident thread obtains the fragment number of the target data fragment to be processed from the fragment control table, and obtains a plurality of target data with the same corresponding fragment number as the fragment number of the target data fragment from the quick access table according to the fragment number of the target data fragment, wherein the target data fragment comprises a plurality of target data which are data corresponding to the same hot spot field in the database, and the control target resident thread sequentially processes the plurality of target data. The method has the following technical effects: the target resident thread sequentially processes a plurality of target data in a sequential execution mode, the complexity of the data processing process is low, the performance damage degree is low, and meanwhile, the hot spot field can be updated according to expectations.
Description
Technical Field
Embodiments of the present invention relate to the field of communications technologies, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
In the course of operating a database, data in certain fields need to be updated frequently concurrently, and these fields may be referred to as hot spot fields. If a plurality of received data are updated for a certain hot spot field at the same time, how to update the hot spot field is very important.
Currently, when a plurality of data corresponding to the same hot spot field are received at the same time, the hot spot field is updated in parallel according to the data.
However, the above-mentioned update procedure may lead to a high complexity of the update procedure and a high performance loss, on the one hand, and on the other hand, the parallel update may lead to that the hot spot field cannot be updated as expected.
Disclosure of Invention
The invention provides a data processing method, a device, equipment and a storage medium, which are used for solving the technical problems that the complexity is high, the performance loss is high and the hot spot field cannot be updated according to expectations when the hot spot field is updated at present.
In a first aspect, an embodiment of the present invention provides a data processing method, including:
the control target resident thread obtains the fragment number of the target data fragment to be processed from the fragment control table;
controlling the target resident thread to acquire a plurality of target data with the same corresponding fragment numbers as the fragment numbers of the target data fragments from a quick access table according to the fragment numbers of the target data fragments; the target data fragments comprise the plurality of target data, wherein the plurality of target data are data corresponding to the same hot spot field in the database;
and controlling the target resident thread to sequentially process a plurality of target data.
In a second aspect, an embodiment of the present invention provides a data processing apparatus, including:
the first acquisition module is used for controlling the target resident thread to acquire the fragment number of the target data fragment to be processed from the fragment control table;
the second acquisition module is used for controlling the target resident thread to acquire a plurality of target data with the same corresponding fragment numbers as the fragment numbers of the target data fragments from the quick access table according to the fragment numbers of the target data fragments; the target data fragments comprise the plurality of target data, wherein the plurality of target data are data corresponding to the same hot spot field in the database;
and the processing module is used for controlling the target resident thread to sequentially process a plurality of target data.
In a third aspect, an embodiment of the present invention further provides a computer apparatus, including:
one or more processors;
a memory for storing one or more programs;
the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data processing method as provided in the first aspect.
In a fourth aspect, embodiments of the present invention also provide a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a data processing method as provided in the first aspect.
The embodiment provides a data processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: the control target resident thread obtains the fragment number of the target data fragment to be processed from the fragment control table, and obtains a plurality of target data with the same corresponding fragment number as the fragment number of the target data fragment from the quick access table according to the fragment number of the target data fragment, wherein the target data fragment comprises a plurality of target data which are data corresponding to the same hot spot field in the database, and the control target resident thread sequentially processes the plurality of target data. The method has the following technical effects: compared with the situation that the update process is higher in complexity and higher in performance loss caused by processing a plurality of data corresponding to the same hot spot field in a parallel mode at present and the hot spot field cannot be updated according to expectations, the data processing method provided by the embodiment avoids processing the plurality of target data in the parallel mode, is low in complexity and low in performance damage degree, and can realize that the hot spot field is updated according to expectations.
Drawings
FIG. 1 is a flow chart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a schematic illustration of interactions between tables in an embodiment of the invention;
FIG. 3 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting thereof. It should be further noted that, for convenience of description, only some, but not all of the structures related to the present invention are shown in the drawings.
Fig. 1 is a flow chart of a data processing method according to an embodiment of the invention. The embodiment is suitable for a scene of updating the hot spot field in the database. The present embodiments may be performed by data processing means, which may be implemented in software and/or hardware, which may be integrated in a computer device. As shown in fig. 1, the data processing method provided in this embodiment includes the following steps:
step 101: and the control target resident thread acquires the fragment number of the target data fragment to be processed from the fragment control table.
Specifically, in this embodiment, a plurality of resident threads are provided, and the number of resident threads may be specified in the yml configuration file.
The slice control table in this embodiment stores slice numbers of each data slice to distinguish different data slices. That is, the primary key of the slice control table is the slice number. The data sharding in this embodiment refers to a collection of data used to update the same hotspot fields in the database. The hot spot field in this embodiment may be a field predefined according to the service scenario and the actual requirement. The hot spot field may be one or more fields such as a "status", "progress", "timestamp of the job", "error code", and "error description" field.
And the plurality of resident threads acquire the fragment numbers of the target data fragments to be processed from the fragment control table in a competitive mode. In this embodiment, for convenience of description, a thread currently competing for data fragmentation is referred to as a target thread, and a currently competing data fragmentation is referred to as a target data fragmentation.
The target data fragment in the present embodiment includes a plurality of target data. That is, the target data fragment in the present embodiment is substantially a set of a plurality of target data. The plurality of target data are data corresponding to the same hot spot field in the database, that is, the plurality of target data are data used for updating the same hot spot field in the database.
The following describes in detail how the target resident thread obtains the fragment number of the target data fragment.
In one implementation, the slice control table further includes a lock flag. Wherein a fragment number corresponds to a lock flag. The lock flag is used to identify whether the data fragment is currently being processed. When the lock flag is in a locked state, indicating that the data slice is currently being processed; when the lock flag is in the released state, it indicates that the data slice is not currently being processed.
In the implementation mode, a control target resident thread acquires a lock mark of a first data fragment from a fragment control table; and if the lock mark of the first data fragment is in a release state, controlling the target resident thread to determine the first data fragment as the target data fragment, and determining the fragment number of the first data fragment as the fragment number of the target data fragment. The first data slice in this embodiment may refer to any one of the data slices in the slice control table. In the above process, the control target resident thread acquires the lock flag of any data slice (called the first data slice for convenience of description) from the slice control table, and if the lock flag is in a release state, it indicates that the first data slice is not currently processed. And controlling the resident thread to take the first data slice as a target data slice and take the slice code of the first data slice as the slice code of the target data slice.
Still further, based on the above implementation, the slice control table further includes: token and execution start time. A fragment number corresponds to a lock flag, a token, and an execution start time. After the control target resident thread sequentially processes the target data, the control target resident thread modifies a lock mark of the target data fragment in the fragment control table into a locking state so as to indicate that the target data fragment is currently being processed; the control target resident thread updates the token of the target data fragment in the fragment control table into the token of the target resident thread to indicate that the target data fragment is currently being processed by the target resident thread; after the target resident thread starts to process the target data in sequence, the target resident thread is controlled to update the execution start time of the target data fragment in the fragment control table so as to indicate when the target data fragment is processed. There is no timing relationship between the steps of modifying the lock flag, updating the token, and updating the execution start time.
In another implementation manner, in the process of acquiring the fragment number of the target data fragment, if the lock flag of the first data fragment is in a locked state, determining whether the absolute value of the difference value between the execution start time and the current time of the first data fragment is greater than a preset time threshold; if the absolute value of the difference value between the execution starting time and the current time of the first data slice is larger than the preset time threshold, determining the first data slice as a target data slice, and determining the slice number of the first data slice as the slice number of the target data slice; the control target resident thread updates the token of the target data fragment in the fragment control table into the token of the target resident thread; after the target resident thread starts to sequentially process the target data, the target resident thread is controlled to update the execution starting time of the target data fragments in the fragment control table.
In this implementation manner, the lock flag of the first data slice is in a locked state, and the absolute value of the difference between the execution start time and the current time of the first data slice is greater than the preset time threshold, which indicates that the processing time of the first data slice is overtime although the first data slice is currently being processed, that is, the processing process is overtime due to abnormal conditions such as faults and the like of the resident thread that previously processes the first data slice. In order to enable the target data fragments to be continuously executed, the target resident thread determines the first data fragments as target data fragments, determines the fragment numbers of the first data fragments as the fragment numbers of the target data fragments, and updates the tokens of the target data fragments in the fragment control table as the tokens of the target resident thread; after the target resident thread starts to sequentially process the target data, the target resident thread is controlled to update the execution starting time of the target data fragments in the fragment control table.
Further, the present embodiment may further include the steps of: controlling a target resident thread to judge whether a token of the target data fragment is a token of the target resident thread or not in the process of sequentially processing target data; and if the token of the target data fragment is not the token of the target resident thread, controlling the target resident thread to stop processing the target data. In the process of sequentially processing the target data fragments, the target resident thread can judge whether the tokens of the target data fragments are tokens of the target resident thread or not at preset time intervals. If the token of the target data fragment is not the token of the target resident thread, which indicates that the target data fragment is processed by other resident threads, the target resident thread is controlled to stop processing the target data. So as to avoid the repeated processing of the target data fragments and save processing resources.
In still another implementation manner, in the process of obtaining the slice number of the target data slice, if the lock flag of the first data slice is in a locked state and the absolute value of the difference between the execution start time and the current time of the first data slice is less than or equal to the preset time threshold, taking the other data slices different from the first data slice in the slice control table as new first data slices, and repeating the process of obtaining the slice number of the target data slice until the slice number of the target data slice is determined.
Step 102: and the control target resident thread acquires a plurality of target data with the same corresponding fragment numbers as the fragment numbers of the target data fragments from the quick access table according to the fragment numbers of the target data fragments.
The target data fragments comprise a plurality of target data, and the target data are data corresponding to the same hot spot field in the database.
Specifically, the quick access table in this embodiment stores unprocessed data. Further, the quick access table may further store a fragment number, a database clock, a module deployment name, and a local sequence number corresponding to the unprocessed data. The primary key of the quick access table may be: fragment number, database clock, module deployment name, and local sequence number. The target data in this embodiment is part of the unprocessed data.
The following describes in detail how the target data is stored in the quick access table. The target data in the present embodiment is transmitted to the data processing apparatus by the transmitting apparatus. The data processing device receives a plurality of target data sent by the sending device, determines the fragment number of the target data according to the mapping relation between the identification of the data and the fragment number and the identification of the target data, and stores the target data in the position corresponding to the fragment number of the target data in the quick access table. The transmitting device in this embodiment may be a device that generates target data. In this embodiment, the fragment numbers of the plurality of target data are the same.
It should be noted that, when storing the target data in the location corresponding to the fragment number of the target data in the quick access table, since the plurality of target data corresponds to one fragment number, each target data may be stored in the location corresponding to the target data in combination with the database clock, the module deployment name and the local sequence number.
In this embodiment, the identification of the target data may be a hotspot key field or a job number of the target data.
In one implementation, a mapping relation between a hot spot key field or a job number and a fragment number of data is configured in advance, after receiving a target data, the fragment number corresponding to the target data is determined from the mapping relation according to the hot spot key field or the job number of the target data, and then the target data is stored in a position corresponding to the fragment number in a quick access table.
In another implementation manner, a mapping relationship between a value obtained by taking the hash value of the hot key field or the job number of the data and the fragment number is configured in advance, after receiving a target data, the value obtained by taking the hash value of the hot key field or the job number of the target data is determined, the fragment number corresponding to the target data is determined from the mapping relationship, and then the target data is stored in a position corresponding to the fragment number in the quick access table. Specifically, a hash algorithm built in a java standard edition development packet (Java SE Development kit, JDK) can be used to calculate a hash value, then a value obtained after the hash value is obtained is determined, and the fragment number corresponding to the target data is determined according to the value obtained after the hash value is obtained.
The number of bits of the fragment number can be set according to actual requirements. Alternatively, the slice number may be 4 bits.
In this embodiment, the transmitting apparatus may transmit the target data through a kafka middleware. The transmitting device may transmit data (these data include target data) to the data processing apparatus in batch through the kafka middleware. Correspondingly, the data processing apparatus in the present embodiment receives a plurality of target data transmitted by the transmitting device through the kafka middleware. Upon specific reception, the data processing apparatus locates specific data in the kafka middleware based on the current data index in the kafka middleware, and acquires the data.
Further, a water meter is also provided in this embodiment. All data sent to the data processing device by the sending equipment are recorded in the flow water meter, so that the follow-up check is convenient, and the data is prevented from being missed.
The data processing device stores the plurality of target data in the flow meter by adopting an increasing operation after receiving the plurality of target data so as to compare the data in the flow meter with the processed data. Wherein the adding operation is an adding operation according to a main key single bar of the flow water meter. In this embodiment, the adding operation of the flow meter is a "micro" operation of adding according to the main key single bar of the flow meter, and compared with the batch operation mode, a large number of concurrent operation databases can be avoided. The conventional operation of adding a plurality of data can be performed, and the plurality of data can be operated, so that the archive log of the database can be rapidly increased due to the mutual nesting of the plurality of data, and even the database cannot be accessed after the disk space is fully filled.
After storing the target data in the streaming water meter, the data processing apparatus modifies the current data index in the kafka middleware to the index of the data in the next batch of data in the kafka middleware that has not yet been transmitted, and commits the kafka transaction.
In this embodiment, after the control target resident thread obtains the fragment number of the target data fragment, a plurality of target data with the same corresponding fragment number as the fragment number of the target data fragment are obtained from the fast access table.
Step 103: and controlling the target resident thread to sequentially process the plurality of target data.
Specifically, after the target resident thread acquires a plurality of target data, the target resident thread may sequentially analyze the plurality of target data according to a sequence defined by the target resident thread.
In one implementation, the quick access table further includes a generation timestamp of the target data. One possible data processing procedure is shown below, based on this implementation.
The control target resident thread analyzes the first target data and obtains the generation time stamp of the target data. And comparing the time stamp of the current value in the hot spot field corresponding to the target data in the database. And if the timestamp of the current value in the hot spot field corresponding to the target data is earlier than the generation timestamp of the target data, updating the value of the hot spot field corresponding to the target data into the target data. And if the timestamp of the current value in the hot spot field corresponding to the target data is later than the generation timestamp of the target data, indicating that the target data is out-of-date data, and discarding the target data.
After the first target data is processed, the target resident thread is controlled to process the second target data according to the implementation mode until all the target data are processed, namely the target data slicing processing is completed. It should be noted that, the order in which the target data is processed by the target resident thread may be predefined.
At present, processing multiple data corresponding to the same hot spot field in a parallel manner can result in higher complexity and higher performance loss of an updating process, and in a situation that the hot spot field cannot be updated as expected, for example, the updating result is unknown when SQL update statement is executed concurrently. In this embodiment, the target resident thread processes multiple target data in a sequential execution manner, which avoids processing multiple target data in a parallel manner, and has low complexity and low performance damage in the data processing process, and meanwhile, can realize that the hot spot field corresponding to the target data is updated according to the expectation.
After the control target resident thread processes the plurality of target data in sequence, deleting the plurality of target data from the quick access table by adopting a deleting operation. The deleting operation is an operation of deleting according to the main key single bar in the quick access table.
The processed target data is deleted, so that the fast access table can maintain better performance, fast writing and fast reading are realized, and the data processing speed of the embodiment is improved.
In this embodiment, the deletion operation is a "micro" operation of deleting according to the primary key single bar in the quick access table, and compared with a batch operation mode, a large number of concurrent operation databases can be avoided. The conventional operation of deleting a plurality of data can be performed, and a plurality of data can be operated, so that the archive log of the database can be rapidly increased due to the mutual nesting of the plurality of data, and even the database cannot be accessed after the disk space is fully filled.
Further, in order to facilitate monitoring of the processing procedure of the target resident thread, in this embodiment, the target resident thread is controlled to send the target data currently being processed to the current state table in the process of processing the target data. The current state table may be displayed on a screen to facilitate monitoring of the data processing process.
In summary, the data processing procedure of the present embodiment involves a plurality of tables: flow meter, quick access table, fragmentation control table and current state table. FIG. 2 is a schematic illustration of interactions between tables in an embodiment of the invention. As shown in fig. 2, all data transmitted from the transmitting device to the data processing apparatus in the present embodiment are stored in the flow meter for later collation. The fast access table stores unprocessed data, and after the target data is processed by the target resident thread, the processed target data is deleted from the fast access table. After the target resident thread obtains the fragment number of the target data fragment from the fragment control table, a plurality of target data with the same corresponding fragment number as the fragment number of the target data fragment are obtained from the quick access table. In the process of controlling the target resident thread to sequentially process the plurality of target data, the target data currently being processed can be displayed in the current state table.
The data processing method provided in this embodiment includes: the control target resident thread obtains the fragment number of the target data fragment to be processed from the fragment control table, and obtains a plurality of target data with the same corresponding fragment number as the fragment number of the target data fragment from the quick access table according to the fragment number of the target data fragment, wherein the target data fragment comprises a plurality of target data which are data corresponding to the same hot spot field in the database, and the control target resident thread sequentially processes the plurality of target data. The method has the following technical effects: compared with the situation that the update process is higher in complexity and higher in performance loss caused by processing a plurality of data corresponding to the same hot spot field in a parallel mode at present and the hot spot field cannot be updated according to expectations, the data processing method provided by the embodiment avoids processing the plurality of target data in the parallel mode, is low in complexity and low in performance damage degree, and can realize that the hot spot field is updated according to expectations.
Fig. 3 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention. As shown in fig. 3, the data processing apparatus provided in this embodiment includes: a first acquisition module 31, a second acquisition module 32 and a processing module 33.
The first obtaining module 31 is configured to control the target resident thread to obtain the fragment number of the target data fragment to be processed from the fragment control table.
The second obtaining module 32 is configured to control the target resident thread to obtain, from the fast access table, a plurality of target data with the corresponding fragment numbers identical to the fragment numbers of the target data fragments according to the fragment numbers of the target data fragments.
The target data fragments comprise a plurality of target data, and the target data are data corresponding to the same hot spot field in the database.
The processing module 33 is configured to control the target resident thread to sequentially process the plurality of target data.
In one implementation, the slice control table further includes: a lock flag. The first obtaining module 31 is specifically configured to: the control target resident thread acquires a lock mark of the first data fragment from the fragment control table; and if the lock mark of the first data fragment is in a release state, controlling the target resident thread to determine the first data fragment as the target data fragment, and determining the fragment number of the first data fragment as the fragment number of the target data fragment.
In another implementation, the slice control table further includes: token and execution start time. The apparatus further comprises: the device comprises a modification module, a first updating module and a second updating module.
The modifying module is used for controlling the target resident thread to modify the lock mark of the target data fragment in the fragment control table into a locking state.
The first updating module is used for controlling the target resident thread to update the token of the target data fragment in the fragment control table into the token of the target resident thread.
And the second updating module is used for controlling the execution starting time of the target data fragments in the target residence updating fragment control table after the target residence thread starts to sequentially process the target data.
In yet another implementation, the apparatus further includes: the device comprises a first determining module, a second determining module, a third updating module and a fourth updating module.
The first determining module is configured to determine whether an absolute value of a difference between an execution start time and a current time of the first data slice is greater than a preset time threshold if the lock flag of the first data slice is in a locked state.
And the second determining module is used for determining the first data slice as the target data slice and determining the slice number of the first data slice as the slice number of the target data slice if the absolute value of the difference value between the execution starting time and the current time of the first data slice is greater than the preset time threshold.
And the third updating module is used for controlling the target resident thread to update the token of the target data fragment in the fragment control table into the token of the target resident thread.
And the fourth updating module is used for controlling the execution starting time of the target data fragments in the target residence updating fragment control table after the target residence thread starts to sequentially process the target data.
In yet another implementation, the apparatus further includes: and the judging module and the stopping processing module.
The judging module is used for controlling the target resident thread to judge whether the token of the target data fragment is the token of the target resident thread or not in the process of sequentially processing the target data.
And the stopping processing module is used for controlling the target resident thread to stop processing the target data if the token of the target data fragment is not the token of the target resident thread.
Optionally, the apparatus further comprises a deletion module.
And the deleting module is used for deleting a plurality of target data from the quick access table by adopting a deleting operation after the target resident threads are controlled to sequentially process the target data. The deleting operation is an operation of deleting according to the main key single bar in the quick access table.
Optionally, the apparatus further comprises: the device comprises a receiving module, a third determining module and a storage module.
And the receiving module is used for receiving the plurality of target data sent by the sending equipment.
And the third determining module is used for determining the fragment number of the target data according to the mapping relation between the identification of the data and the fragment number and the identification of the target data.
And the storage module is used for storing the target data in a position corresponding to the fragment number of the target data in the quick access table.
More specifically, the receiving module is specifically configured to: receiving a plurality of target data transmitted by a transmitting device through kafka middleware; multiple target data are stored in the flow meter using an increment operation to facilitate comparison of the processed data with data in the flow meter. Wherein the adding operation is an adding operation according to a main key single bar of the flow water meter.
The data processing device provided by the embodiment of the invention can execute the data processing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
Fig. 4 is a schematic structural diagram of a computer device according to an embodiment of the present invention. As shown in fig. 4, the computer device includes a processor 40 and a memory 41. The number of processors 40 in the computer device may be one or more, one processor 40 being taken as an example in fig. 4; the processor 40 and the memory 41 of the computer device may be connected by a bus or otherwise, for example in fig. 4.
The memory 41 is a computer-readable storage medium that can be used to store a software program, a computer-executable program, and modules, such as program instructions and modules corresponding to the data processing method in the embodiment of the present invention (for example, the first acquisition module 31, the second acquisition module 32, and the processing module 33 in the data processing apparatus). The processor 40 executes various functional applications of the computer device and data processing, i.e., implements the data processing method described above, by running software programs, instructions and modules stored in the memory 41.
The memory 41 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, at least one application program required for functions; the storage data area may store data created according to the use of the computer device, etc. In addition, memory 41 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 41 may further comprise memory located remotely from processor 40, which may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The present invention also provides a storage medium containing computer executable instructions which, when executed by a computer processor, are for performing a data processing method comprising:
the control target resident thread obtains the fragment number of the target data fragment to be processed from the fragment control table;
controlling the target resident thread to acquire a plurality of target data with the same corresponding fragment numbers as the fragment numbers of the target data fragments from a quick access table according to the fragment numbers of the target data fragments; the target data fragments comprise the plurality of target data, wherein the plurality of target data are data corresponding to the same hot spot field in the database;
and controlling the target resident thread to sequentially process a plurality of target data.
Of course, the storage medium containing computer executable instructions provided in the embodiments of the present invention is not limited to the method operations described above, and may also perform related operations in the data processing method provided in any embodiment of the present invention.
From the above description of embodiments, it will be clear to a person skilled in the art that the present invention may be implemented by means of software and necessary general purpose hardware, but of course also by means of hardware, although in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, etc., and include several instructions for causing a computer device (which may be a personal computer, a computer device, or a network device, etc.) to execute the data processing method according to the embodiments of the present invention.
It should be noted that, in the above embodiment of the data processing apparatus, each unit and module included are only divided according to the functional logic, but not limited to the above division, so long as the corresponding functions can be implemented; in addition, the specific names of the functional units are also only for distinguishing from each other, and are not used to limit the protection scope of the present invention.
Note that the above is only a preferred embodiment of the present invention and the technical principle applied. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, while the invention has been described in connection with the above embodiments, the invention is not limited to the embodiments, but may be embodied in many other equivalent forms without departing from the spirit or scope of the invention, which is set forth in the following claims.
Claims (8)
1. A method of data processing, comprising:
the control target resident thread obtains the fragment number of the target data fragment to be processed from the fragment control table;
controlling the target resident thread to acquire a plurality of target data with the same corresponding fragment numbers as the fragment numbers of the target data fragments from a quick access table according to the fragment numbers of the target data fragments; the target data fragments comprise the plurality of target data, wherein the plurality of target data are data corresponding to the same hot spot field in the database; the quick access table includes a generation timestamp of the target data;
controlling the target resident thread to sequentially process a plurality of target data, including: controlling the target resident thread to analyze first target data and acquiring a generation time stamp of the first target data; comparing the time stamp of the current value in the hot spot field corresponding to the first target data in the database; if the timestamp of the current value in the hot spot field corresponding to the first target data is earlier than the generation timestamp of the first target data, updating the value of the hot spot field corresponding to the first target data into the first target data; discarding the first target data if the timestamp of the current value in the hot spot field corresponding to the first target data is later than the generation timestamp of the first target data; after the first target data is processed, controlling the target resident thread to process second target data according to the mode of processing the first target data until all the target data are processed;
before the control target resident thread obtains the fragment number of the target data fragment to be processed from the fragment control table, the method further comprises: receiving a plurality of target data transmitted by a transmitting device; determining the fragment number of the target data according to the mapping relation between the identification of the data and the fragment number and the identification of the target data; storing the target data in a position corresponding to the fragment number of the target data in the quick access table;
the receiving and transmitting device transmits a plurality of target data, including: receiving a plurality of target data transmitted by the transmitting device through a kafka middleware; storing a plurality of target data in a flow meter by adopting an increasing operation so as to compare the data in the flow meter with processed data; and the adding operation is performed according to a main key single bar of the flow meter.
2. The method of claim 1, wherein the slice control table further comprises: a lock flag;
the control target resident thread obtains the fragment number of the target data fragment to be processed from the fragment control table, and the control target resident thread comprises the following steps:
controlling the target resident thread to acquire a lock mark of a first data fragment from the fragment control table;
and if the lock mark of the first data fragment is in a release state, controlling the target resident thread to determine the first data fragment as the target data fragment, and determining the fragment number of the first data fragment as the fragment number of the target data fragment.
3. The method of claim 2, wherein the slice control table further comprises: token and execution start time;
after the target resident thread is controlled to sequentially process the target data, the method further comprises the following steps:
controlling the target resident thread to modify the lock mark of the target data fragment in the fragment control table into a locking state;
controlling the target resident thread to update the token of the target data fragment in the fragment control table to the token of the target resident thread;
and after the target resident thread starts to sequentially process the target data, controlling the target resident to update the execution starting time of the target data fragments in the fragment control table.
4. A method according to claim 3, characterized in that the method further comprises:
if the lock mark of the first data fragment is in a locking state, determining whether the absolute value of the difference value between the execution starting time and the current time of the first data fragment is larger than a preset time threshold;
if the absolute value of the difference value between the execution starting time and the current time of the first data slice is larger than the preset time threshold, determining the first data slice as the target data slice, and determining the slice number of the first data slice as the slice number of the target data slice;
controlling the target resident thread to update the token of the target data fragment in the fragment control table to the token of the target resident thread;
after the target resident thread starts to sequentially process the target data, controlling the target resident to update the execution starting time of the target data fragments in the fragment control table;
controlling the target resident thread to judge whether the token of the target data fragment is the token of the target resident thread or not in the process of sequentially processing the target data;
and if the token of the target data fragment is not the token of the target resident thread, controlling the target resident thread to stop processing the target data.
5. The method according to any one of claims 1-4, further comprising:
after the target resident thread is controlled to sequentially process and finish the target data, deleting the plurality of target data from the quick access table by adopting a deleting operation; the deleting operation is an operation of deleting according to a main key single bar in the quick access table.
6. A data processing apparatus, comprising:
the first acquisition module is used for controlling the target resident thread to acquire the fragment number of the target data fragment to be processed from the fragment control table;
the second acquisition module is used for controlling the target resident thread to acquire a plurality of target data with the same corresponding fragment numbers as the fragment numbers of the target data fragments from the quick access table according to the fragment numbers of the target data fragments; the target data fragments comprise the plurality of target data, wherein the plurality of target data are data corresponding to the same hot spot field in the database; the quick access table includes a generation timestamp of the target data;
the processing module is used for controlling the target resident thread to sequentially process a plurality of target data;
the processing module is specifically configured to: controlling the target resident thread to analyze first target data and acquiring a generation time stamp of the first target data; comparing the time stamp of the current value in the hot spot field corresponding to the first target data in the database; if the timestamp of the current value in the hot spot field corresponding to the first target data is earlier than the generation timestamp of the first target data, updating the value of the hot spot field corresponding to the first target data into the first target data; discarding the first target data if the timestamp of the current value in the hot spot field corresponding to the first target data is later than the generation timestamp of the first target data; after the first target data is processed, controlling the target resident thread to process second target data according to the mode of processing the first target data until all the target data are processed;
the apparatus further comprises: the device comprises a receiving module, a third determining module and a storage module, wherein the receiving module is used for receiving a plurality of target data sent by a sending device, the third determining module is used for determining the fragment numbers of the target data according to the mapping relation between the identifications of the data and the fragment numbers and the identifications of the target data, and the storage module is used for storing the target data in the positions corresponding to the fragment numbers of the target data in the quick access table;
the receiving module is specifically configured to: receiving a plurality of target data transmitted by the transmitting device through a kafka middleware; storing a plurality of target data in a flow meter by adopting an increasing operation so as to compare the data in the flow meter with processed data; and the adding operation is performed according to a main key single bar of the flow meter.
7. A computer device, the computer device comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, causes the one or more processors to implement the data processing method of any of claims 1-5.
8. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements a data processing method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010242018.8A CN111352948B (en) | 2020-03-31 | 2020-03-31 | Data processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010242018.8A CN111352948B (en) | 2020-03-31 | 2020-03-31 | Data processing method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111352948A CN111352948A (en) | 2020-06-30 |
| CN111352948B true CN111352948B (en) | 2023-12-26 |
Family
ID=71193089
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010242018.8A Active CN111352948B (en) | 2020-03-31 | 2020-03-31 | Data processing method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111352948B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108038239A (en) * | 2017-12-27 | 2018-05-15 | 中科鼎富（北京）科技发展有限公司 | A kind of heterogeneous data source method of standardization management, device and server |
| CN108874552A (en) * | 2018-06-28 | 2018-11-23 | 杭州云英网络科技有限公司 | Distributed lock executes method, apparatus and system, application server and storage medium |
| CN109002484A (en) * | 2018-06-25 | 2018-12-14 | 北京明朝万达科技股份有限公司 | A kind of method and system for sequence consumption data |
| CN109144731A (en) * | 2018-08-31 | 2019-01-04 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer equipment and storage medium |
| CN109491928A (en) * | 2018-11-05 | 2019-03-19 | 深圳乐信软件技术有限公司 | Buffer control method, device, terminal and storage medium |
| CN109976888A (en) * | 2019-03-25 | 2019-07-05 | 苏州达家迎信息技术有限公司 | A kind of data scanning method, device, equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8762378B2 (en) * | 2011-12-23 | 2014-06-24 | Sap Ag | Independent table nodes in parallelized database environments |
-
2020
- 2020-03-31 CN CN202010242018.8A patent/CN111352948B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108038239A (en) * | 2017-12-27 | 2018-05-15 | 中科鼎富（北京）科技发展有限公司 | A kind of heterogeneous data source method of standardization management, device and server |
| CN109002484A (en) * | 2018-06-25 | 2018-12-14 | 北京明朝万达科技股份有限公司 | A kind of method and system for sequence consumption data |
| CN108874552A (en) * | 2018-06-28 | 2018-11-23 | 杭州云英网络科技有限公司 | Distributed lock executes method, apparatus and system, application server and storage medium |
| CN109144731A (en) * | 2018-08-31 | 2019-01-04 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer equipment and storage medium |
| CN109491928A (en) * | 2018-11-05 | 2019-03-19 | 深圳乐信软件技术有限公司 | Buffer control method, device, terminal and storage medium |
| CN109976888A (en) * | 2019-03-25 | 2019-07-05 | 苏州达家迎信息技术有限公司 | A kind of data scanning method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111352948A (en) | 2020-06-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3382534A1 (en) | Page jumping method and apparatus | |
| US10225145B2 (en) | Method and device for updating client | |
| RU2012102818A (en) | METHOD AND DEVICE FOR TESTING A FILE SYSTEM FOR VIRUSES | |
| US8813229B2 (en) | Apparatus, system, and method for preventing infection by malicious code | |
| CN108363579B (en) | Data burning method, device, equipment and storage medium | |
| WO2020232878A1 (en) | Spring mvc-based timed task processing method, apparatus and computer device | |
| CN113132267B (en) | Distributed system, data aggregation method and computer readable storage medium | |
| CN111352948B (en) | Data processing method, device, equipment and storage medium | |
| CN107885634B (en) | Method and device for processing abnormal information in monitoring | |
| CN113132356A (en) | UDP (user Datagram protocol) message distribution method, equipment and storage medium | |
| CN111752819B (en) | Abnormality monitoring method, device, system, equipment and storage medium | |
| CN111385613B (en) | Television system repairing method, storage medium and application server | |
| CN111367869A (en) | Mirror image file processing method and device, storage medium and electronic equipment | |
| CN107872493B (en) | Information processing method, terminal and server | |
| CN109684361B (en) | Rule analysis method and corresponding rule analysis device | |
| CN112817620B (en) | Controller terminal program updating method, device, computer equipment and storage medium | |
| CN112463786B (en) | Data synchronization method, system, server and storage medium | |
| CN106897325B (en) | Data loading method and device | |
| CN110597537B (en) | Safe updating and upgrading method for nodes of Internet of things | |
| CN111224756B (en) | Method and device for determining data transmission abnormity, storage medium and electronic equipment | |
| CN115809150A (en) | Distributed deadlock detection method and device and electronic equipment | |
| WO2013000287A1 (en) | Method and device for configuring cell parameters | |
| CN117407369B (en) | Operation log recording method and device and electronic equipment | |
| KR20120031963A (en) | Apparatus for preventing malicious codes | |
| CN107484150B (en) | EV-DO network connection method, device and terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220914 Address after: 25 Financial Street, Xicheng District, Beijing 100033 Applicant after: CHINA CONSTRUCTION BANK Corp. Address before: 25 Financial Street, Xicheng District, Beijing 100033 Applicant before: CHINA CONSTRUCTION BANK Corp. Applicant before: Jianxin Financial Science and Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |