CN112541041A - Data processing method, device, server and storage medium - Google Patents
Data processing method, device, server and storage medium Download PDFInfo
- Publication number
- CN112541041A CN112541041A CN202011545402.1A CN202011545402A CN112541041A CN 112541041 A CN112541041 A CN 112541041A CN 202011545402 A CN202011545402 A CN 202011545402A CN 112541041 A CN112541041 A CN 112541041A
- Authority
- CN
- China
- Prior art keywords
- task
- target
- fragment
- fragmentation
- processed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 20
- 239000012634 fragment Substances 0.000 claims abstract description 238
- 238000013467 fragmentation Methods 0.000 claims abstract description 185
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 185
- 238000000034 method Methods 0.000 claims abstract description 45
- 230000015654 memory Effects 0.000 claims description 14
- 238000001514 detection method Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 6
- 230000007246 mechanism Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 3
- HRANPRDGABOKNQ-ORGXEYTDSA-N (1r,3r,3as,3br,7ar,8as,8bs,8cs,10as)-1-acetyl-5-chloro-3-hydroxy-8b,10a-dimethyl-7-oxo-1,2,3,3a,3b,7,7a,8,8a,8b,8c,9,10,10a-tetradecahydrocyclopenta[a]cyclopropa[g]phenanthren-1-yl acetate Chemical compound C1=C(Cl)C2=CC(=O)[C@@H]3C[C@@H]3[C@]2(C)[C@@H]2[C@@H]1[C@@H]1[C@H](O)C[C@@](C(C)=O)(OC(=O)C)[C@@]1(C)CC2 HRANPRDGABOKNQ-ORGXEYTDSA-N 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/278—Data partitioning, e.g. horizontal or vertical partitioning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
- G06F9/5066—Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs
Abstract
The embodiment of the application relates to the field of big data and discloses a data processing method, a device, a server and a storage medium, wherein the method comprises the following steps: acquiring a first target task, wherein the first target task belongs to a first to-be-processed subtask; determining to-be-processed data and a data processing mode indicated by a first target task; acquiring task fragmentation information from a database, and acquiring a fragmentation state of a target fragment corresponding to a first target task from the database according to the task fragmentation information; and according to the fragment state of the target fragment, performing data processing on the to-be-processed data indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task. By the method, the data can be processed concurrently, repeated data processing is avoided, and the data processing efficiency is improved. The present invention relates to a blockchain technique, such as writing the data into a blockchain for data forensics and other scenarios.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a data processing method, an apparatus, a server, and a storage medium.
Background
With the rapid development of computer technology, the amount of various service data is also increasing, when service data is processed, data is usually processed in a manner of concurrently executing a plurality of tasks, and concurrent processing may cause repeated processing, thereby affecting data processing efficiency, and therefore, how to process data quickly and efficiently becomes crucial.
Disclosure of Invention
The embodiment of the application provides a data processing method, a data processing device, a server and a storage medium, which can achieve concurrent data processing, avoid repeated data processing and improve data processing efficiency.
In a first aspect, an embodiment of the present application discloses a data processing method, which is applied to a server, where the server is any one of a server cluster, and the method includes:
the method comprises the steps of obtaining a first target task, wherein the first target task belongs to a first to-be-processed subtask, and the first to-be-processed subtask is any one of a plurality of to-be-processed subtasks obtained by dividing the to-be-processed task;
determining to-be-processed data and a data processing mode indicated by the first target task according to the first target task;
acquiring task fragment information from a database, and acquiring a fragment state of a target fragment corresponding to the first target task from the database according to the task fragment information;
and according to the fragment state of the target fragment, performing data processing on the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
In a second aspect, an embodiment of the present application discloses a data processing apparatus, including:
the device comprises a first acquisition unit, a first processing unit and a second acquisition unit, wherein the first acquisition unit is used for acquiring a first target task, the first target task belongs to a first to-be-processed subtask, and the first to-be-processed subtask is any one of a plurality of to-be-processed subtasks obtained by dividing the to-be-processed task;
the determining unit is used for determining the data to be processed and the data processing mode indicated by the first target task according to the first target task;
the second acquisition unit is used for acquiring task fragment information from a database and acquiring the fragment state of a target fragment corresponding to the first target task from the database according to the task fragment information;
and the processing unit is used for performing data processing on the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task according to the fragment state of the target fragment.
In a third aspect, an embodiment of the present application discloses a server, including a processor, a memory, and a network interface, where the processor, the memory, and the network interface are connected to each other, where the memory is used to store a computer program, and the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method of the first aspect.
In a fourth aspect, an embodiment of the present application discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program includes program instructions, which, when executed by a processor, cause the processor to execute the method of the first aspect.
In the embodiment of the application, a server can obtain a first target task, wherein the first target task belongs to a first to-be-processed subtask, the first to-be-processed subtask is any one of a plurality of to-be-processed subtasks obtained by dividing the to-be-processed task, then, the server can determine to-be-processed data and a data processing mode indicated by the first target task according to the first target task, obtain task fragmentation information from a database, and obtain a fragmentation state of a target fragmentation corresponding to the first target task from the database according to the fragmentation information; further, the server may perform data processing on the to-be-processed data indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task according to the fragment state of the target fragment. It can be seen that the task is divided into a plurality of subtasks, the plurality of subtasks can be processed concurrently, and whether to perform data processing can be determined according to the acquired fragment state of the fragment corresponding to the task, so that repeated data processing can be avoided, and the efficiency of data processing can be improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flowchart of a data processing method provided in an embodiment of the present application;
FIG. 2 is a schematic flow chart diagram of another data processing method provided in the embodiments of the present application;
FIG. 3 is a schematic flow chart diagram illustrating a further data processing method provided in an embodiment of the present application;
fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a server according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a data processing method according to an embodiment of the present disclosure. The data processing method described in this embodiment is applied to a server, where the server is any one of a server cluster, and the data processing method includes the following steps:
s101: and acquiring a first target task.
In one implementation, a server may obtain a first target task. The first target task belongs to a first to-be-processed sub-task, the first target task comprises a part of or all of the first to-be-processed sub-task, and the first to-be-processed sub-task is any one of a plurality of to-be-processed sub-tasks obtained by dividing the to-be-processed task. For example, the task to be processed may refer to a certain type of operation, and if the payment period for the policy is updated, the number of policies is huge, and there may be ten million policies that need to be processed, the operation needs to be updated simultaneously for the payment period corresponding to the ten million policies, that is, the task to be processed is to update all the payment periods of the ten million policies. The first pending subtask may be to update all of the repayment sessions for one of the one million policies and the first target task may be to update all of the repayment sessions for one of the one million policies. In this way, the policy with large data volume is divided into a plurality of shares so as to concurrently process the updating operation of the corresponding period of the policy.
In one implementation, the server is any one of a server cluster, and in the data processing system, a plurality of servers may be configured, that is, one server cluster is configured, so that a plurality of servers are used to process a plurality of to-be-processed sub-tasks of the to-be-processed tasks at the same time. It should be noted that, one server may obtain a part of tasks or all tasks in one to-be-processed sub-task, and the server may determine to obtain the size of the task data volume in one to-be-processed sub-task according to its own idle load volume. If one server acquires a part of tasks in a to-be-processed subtask, another server can acquire other tasks except the part of tasks in the to-be-processed subtask. For example, the number of the to-be-processed subtasks is 5, the number of the servers is 3, each server can acquire part of or all of the tasks (the acquired tasks are different) in any one of the to-be-processed subtasks, and after the task corresponding to a certain server is processed, the remaining to-be-processed subtasks can be acquired. When the number of the servers is greater than or equal to the number of the sub-tasks to be processed, the same understanding is carried out, and the description is omitted here.
For example, the pending task is to update all repayment periods of ten million insurance policies, and the pending task is divided into 5 pending subtasks, assuming that the dividing is average, that is, each pending subtask is to update all repayment periods of two million insurance policies in the ten million insurance policies. Then, 3 servers may respectively obtain a part of or all of the 5 to-be-processed subtasks, for example, a first server may obtain a part of or all of the first to-be-processed subtask, a second server may obtain the remaining tasks in the first to-be-processed subtask, or may obtain a part of or all of the other to-be-processed subtasks, and a third server may understand the same.
In one implementation, the number of the servers in the server cluster may also be determined according to the task data amount of the task to be processed or the urgency of the task to be processed. For example, if the amount of task data of the task to be processed is large or the urgency of the task to be processed is high, more servers may be configured. The method and the system have the advantages that more servers are utilized to process tasks to be processed simultaneously, and the data processing efficiency is improved.
In one implementation, the to-be-processed task may be divided into a plurality of to-be-processed subtasks according to a preset division condition. Specifically, the dividing condition includes the number of current servers, the task data amount of the task to be processed, the urgency degree of the task to be processed, and the like, so as to determine the number of the subtasks to be processed according to the dividing condition. For example, if the number of current servers is large, or the amount of task data of the to-be-processed task is large, or the urgency degree of the to-be-processed task is high, the number of the to-be-processed subtasks may also be large, so as to divide the to-be-processed task into a plurality of tasks to be processed simultaneously, thereby improving the efficiency of data processing.
In an implementation manner, after acquiring a first target task, a server may further acquire a task amount of the first target task, and detect whether the task amount of the first target task is less than or equal to an idle load amount of the server, where the idle load amount is a load amount at which the current server can process data. If the detection result is that the task amount of the first target task is less than or equal to the idle load amount of the server, the step S102 is executed, and if the detection result is that the task amount of the first target task is greater than the idle load amount of the server, the first target task may be pushed to other servers.
S102: and determining the data to be processed and the data processing mode indicated by the first target task according to the first target task.
In an implementation manner, the first target task may be used to indicate to-be-processed data and a processing manner, and then when the server processes to-be-processed data corresponding to the first target task, the server may process to-be-processed data according to the processing manner indicated by the first target task. For example, if the to-be-processed data indicated by the first target task is a repayment period of one million insurance policies, and the processing mode is to update the repayment period of the one million insurance policies, the server may perform data processing on the to-be-processed data indicated by the first target task in the target fragments after subsequently acquiring the target fragments corresponding to the first target task, that is, update the repayment period of one million insurance policies.
S103: task fragmentation information is obtained from a database, and the fragmentation state of a target fragment corresponding to the first target task is obtained from the database according to the task fragmentation information.
In one implementation, the database is a shard database, each shard in the database may correspond to each to-be-processed subtask, and the shard number of the database is the number of all to-be-processed subtasks. For example, for all repayment periods for which the pending task is to update a ten million policy, the ten million policy may be divided into 12 shares, that is, 12 pending subtasks, and correspondingly, there are 12 pieces in the database, where each piece corresponds to the pending data indicated by one pending subtask.
In an implementation manner, the database may further include task fragmentation information, where the task fragmentation information includes a correspondence between task identifiers and fragmentation identifiers, and a fragmentation state of each fragmentation in the database. The shard state may include a first state and a second state, where the first state indicates that the shard is available, that is, the shard is currently occupied by no server, that is, no server performs data processing on data in the shard. Correspondingly, the second state indicates that the fragment is unavailable, that is, the fragment is currently occupied by the server, that is, there is a server performing data processing on data in the fragment. Then, before the server acquires the target fragment, it is further required to acquire a fragment state corresponding to the target fragment, and determine whether the acquired fragment state is the first state or the second state, so as to determine whether the server can acquire the target fragment corresponding to the first target task currently according to the fragment state to perform subsequent steps.
In an implementation manner, a first target task acquired by a server carries a task identifier, and the task identifier may be used to indicate a to-be-processed subtask to which the first target task belongs. The server may obtain the task fragment information from the database, and obtain a corresponding relationship between the task identifier and the fragment identifier in the task fragment information, so as to determine, according to the corresponding relationship, the fragment identifier corresponding to the task identifier carried in the first target task. After the fragment identifier corresponding to the task identifier carried in the first target task is determined, the fragment state of the target fragment corresponding to the fragment identifier of the task identifier carried in the first target task can be acquired from the task fragment information according to the fragment identifier.
S104: and according to the fragment state of the target fragment, performing data processing on the to-be-processed data indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
In an implementation manner, the server may perform different operations according to the obtained fragmentation state of the target fragment. If the fragmentation state of the target fragment is the first state, the server may obtain the target fragment corresponding to the fragmentation identifier from the database according to the fragmentation identifier corresponding to the task identifier carried in the first target task, and perform data processing on the to-be-processed data indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task. If the fragmentation state of the target fragmentation is the second state, the server may determine whether to acquire the target fragmentation for data processing according to the acquired fragmentation state of the target fragmentation within a preset time length. And if the fragment state of the target fragment acquired within the preset time length is the first state, acquiring the target fragment corresponding to the fragment identifier from the database, and performing data processing on the to-be-processed data indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task. If the fragmentation state of the target fragmentation acquired within the preset time length is the second state, processing of the first target task can be abandoned, and the second target task can be acquired again, wherein the second target task is the task except the first to-be-processed subtask in the to-be-processed subtask, so that the problem that the waiting time length of the server is too long and the data processing efficiency is affected is avoided.
In the embodiment of the application, a server can obtain a first target task, wherein the first target task belongs to a first to-be-processed subtask, the first to-be-processed subtask is any one of a plurality of to-be-processed subtasks obtained by dividing the to-be-processed task, then, the server can determine to-be-processed data and a data processing mode indicated by the first target task according to the first target task, obtain task fragmentation information from a database, and obtain a fragmentation state of a target fragmentation corresponding to the first target task from the database according to the fragmentation information; further, the server may perform data processing on the to-be-processed data indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task according to the fragment state of the target fragment. Therefore, the task is divided into the plurality of subtasks, so that the server in the server cluster concurrently processes the plurality of subtasks, and whether to perform data processing can be determined according to the acquired fragment state of the fragment corresponding to the task, so that repeated data processing can be avoided, and the data processing efficiency can be improved.
Referring to fig. 2, fig. 2 is a schematic flowchart illustrating another data processing method according to an embodiment of the present disclosure. The data processing method described in this embodiment is applied to a server, where the server is any one of a server cluster, and the data processing method includes the following steps:
s201: and acquiring a first target task.
S202: and determining the data to be processed and the data processing mode indicated by the first target task according to the first target task.
S203: and acquiring task fragmentation information from a database, and acquiring the corresponding relation between the task identifier and the fragmentation identifier from the task fragmentation information.
In one implementation, the database may include two types of data, one is to store data related to task fragments, that is, task fragment information, and the other is to store data to be processed corresponding to tasks to be processed. Optionally, the task fragmentation information may include a corresponding relationship between a task identifier and a fragmentation identifier, where the task identifier or the fragmentation identifier may be a number, a bit value, or other information, and the application is not limited. The task identifier may be used to indicate a specific to-be-processed subtask to which the first target task obtained by the server belongs, the fragment identifier is an identifier of each fragment in the database, and the fragment identifier may be used to indicate a specific corresponding fragment in the database.
S204: and determining the fragment identifier corresponding to the task identifier carried in the first target task according to the corresponding relation between the task identifier and the fragment identifier.
In an implementation manner, the server may first obtain task fragmentation information from the server, and obtain a correspondence between the task identifier and the fragmentation identifier from the task fragmentation information, so as to determine, according to the task identifier carried in the first target task and the correspondence, the fragmentation identifier corresponding to the task identifier carried in the first target task. For example, table 1 shows a corresponding relationship between task identifiers and task identifiers, and as can be seen from table 1, if a task identifier carried in a first target task is 2, a corresponding segment identifier is 2, and if a task identifier carried in the first target task is 3, a corresponding segment identifier is 3.
Table 1:
| task identification | Fragmentation identification |
| 1 | 1 |
| 2 | 2 |
| 3 | 3 |
| 4 | 4 |
S205: and acquiring the fragmentation state of the target fragment corresponding to the fragmentation identifier of the task identifier carried in the first target task from the task fragmentation information according to the fragmentation identifier corresponding to the task identifier carried in the first target task.
In an implementation manner, the task fragmentation information in the database may include, in addition to the correspondence between the task identifier and the fragmentation identifier, a fragmentation state of each fragmentation in the database. The fragmentation state may include a first state and a second state, where the first state indicates that the fragmentation is available, that is, the fragmentation is currently occupied by no server, that is, no server performs data processing on data in the fragmentation. Correspondingly, the second state indicates that the fragment is unavailable, that is, the fragment is currently occupied by the server, that is, the server performs data processing on the data in the fragment. Optionally, the fragmentation status of each fragment may be implemented by using a locking mechanism, for example, the locking mechanism may be a Compare and Swap (CAS) locking mechanism, i.e., a "Compare and Swap" locking mechanism. Specifically, each shard has a corresponding CAS lock, and the CAS lock can be used to lock the data of the shard, so that other servers cannot process the data of the shard, and repeated processing is avoided. The fragmentation state of the CAS lock changes according to whether a server occupies the fragment or not, if the fragment does not occupy the server currently, the fragmentation state of the CAS lock corresponding to the fragment is a first state, and if the fragment occupies the server currently, the fragmentation state of the CAS lock corresponding to the fragment is a second state. Wherein the first state may be represented by a digital "0" and the second state may be represented by a digital "1".
In an implementation manner, after determining a fragment identifier corresponding to a task identifier carried in a first target task, a server may obtain, from task fragment information, a fragment state corresponding to the fragment identifier of the task identifier carried in the first target task, where the fragment state is a fragment state of a target fragment corresponding to the first target task. Optionally, the task segment information stores a corresponding relationship between the segment identifier and the segment state, for example, the corresponding relationship between the segment identifier and the segment state is shown in table 2, in this application, a first state included in the segment state is represented by a number "0", and a second state is represented by a number "1". As can be seen from table 2, if the fragment identifier of the task identifier carried in the first target task, which is obtained by the server, is 1, the corresponding fragment state is 1, which indicates that the target fragment corresponding to the first target task is unavailable, and the current target fragment is occupied by other servers in the server cluster, that is, other servers perform data processing on data in the target fragment; if the fragment identifier of the task identifier carried in the first target task, which is acquired by the server, is 3, the corresponding fragment state is 0, that is, it indicates that the target fragment corresponding to the first target task is available, and the current target fragment is not occupied by other servers in the server cluster, that is, no server performs data processing on data in the target fragment.
Table 2:
| fragmentation identification | Fragmentation status |
| 1 | 1 |
| 2 | 0 |
| 3 | 0 |
| 4 | 1 |
S206: and according to the fragment state of the target fragment, performing data processing on the to-be-processed data indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
In an implementation manner, the server may execute different operations according to the fragmentation state of the target fragment, so as to implement data processing on the to-be-processed data indicated by the first target task in the target fragment by using the data processing manner indicated by the first target task.
In an implementation manner, if the fragmentation state of the target fragmentation is the first state, it indicates that the target fragmentation may be available, and the server may obtain, from the database, the target fragmentation corresponding to the fragmentation identifier corresponding to the task identifier carried in the first target task according to the fragmentation identifier corresponding to the task identifier carried in the first target task. After the target fragment corresponding to the first target task is obtained, data processing can be performed on the to-be-processed data indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
For example, if the to-be-processed data indicated by the first target task is a repayment period of one million insurance policy, and the processing mode is to update the repayment period of the one million insurance policy, after the server acquires the target segment, the to-be-processed data indicated by the first target task in the target segment, that is, the repayment period of the one million insurance policy, may be subjected to data processing, that is, the repayment period of the one million insurance policy is updated.
It should be noted that after the server obtains the target fragment corresponding to the fragment identifier corresponding to the task identifier carried in the first target task from the database, the fragment state of the target fragment needs to be updated from the first state to the second state to indicate that the target fragment is currently occupied, and then other servers in the server cluster cannot obtain the target fragment currently, so that repeated processing of data in the target fragment can be avoided. After the server performs data processing on the to-be-processed data indicated by the first target task in the target fragment by using the data processing manner indicated by the first target task, the fragment state of the target fragment needs to be updated from the second state to the first state, so as to indicate that the target fragment is not occupied currently and is available for other servers to subsequently acquire the target fragment for data processing.
In an implementation manner, if the fragmentation state of the target fragmentation is the second state, it indicates that the target fragmentation may be unavailable, and other servers in the server cluster currently occupy the target fragmentation to perform corresponding data processing, the server may wait at the target fragmentation, and after the other servers finish processing corresponding data by the target fragmentation, perform data processing on the to-be-processed data indicated by the first target task in the target fragmentation by using the data processing manner indicated by the first target task. Specifically, the server may obtain the fragmentation state of the target fragmentation within a preset time length, and determine whether the fragmentation state of the target fragmentation is updated to the first state. Optionally, the server may obtain the fragmentation state of the target fragment from the task fragmentation information according to a preset time interval within a preset time duration, and determine whether the fragmentation state of the target fragment is updated to the first state. And then different operations are executed according to the updated fragment state of the target fragment so as to realize the data processing of the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
In one implementation, if the server determines that the fragmentation status of the target fragmentation is updated to the first status within a preset time length, it indicates that the target fragmentation is currently available. Then, the server may obtain, from the database, the target segment corresponding to the segment identifier corresponding to the task identifier carried in the first target task according to the segment identifier corresponding to the task identifier carried in the first target task. And then after the target fragment corresponding to the first target task is obtained, performing data processing on the to-be-processed data indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
In one implementation, if the server determines that the fragmentation state of the target fragmentation has not been updated to the first state within a preset time length, it indicates that the target fragmentation is currently unavailable. Then, the server may give up processing the first target task and reacquire the second target task, so as to avoid that the waiting time of the server is too long, which causes a task deadlock and affects the efficiency of data processing. The second target task belongs to a second to-be-processed sub-task, the second target task may include a part of or all of the second to-be-processed sub-task, the second to-be-processed sub-task is any one of the plurality of to-be-processed sub-tasks, and the first to-be-processed sub-task is different from the second to-be-processed sub-task. If the server still obtains the second target task as a partial task in the first to-be-processed subtask, the target fragment is still obtained in the subsequent step, that is, the fragment state of the target fragment is in the second state, and the server still needs to wait, so the second target task should not belong to the partial task in the first to-be-processed subtask. After the server acquires the second target task, data to be processed indicated by the second target task may be processed in the target fragment according to the data processing mode indicated by the second target task. It should be understood that the process of the server processing the second target task is the same as the process of the server processing the first target task, and is not described herein.
Optionally, the preset time duration and the preset time interval may be preset based on various conditions, for example, the preset time duration may be 10 seconds, and the preset time interval may be 2 seconds, which is not specifically limited in this embodiment of the present application. The preset time length and the preset time interval can be determined according to the number of the servers in the server cluster, the task data volume of the task to be processed, the emergency degree of the task to be processed and the like. For example, if the number of the current servers is large, the task data amount of the to-be-processed task is large, or the urgency degree of the to-be-processed task is high, the preset time length and the preset time interval may be set to be short, so as to quickly process the remaining to-be-processed tasks, and thus, the influence on the data processing efficiency due to the overlong waiting time length of the servers can be avoided.
For specific implementation of steps S201 to S202, reference may be made to the detailed description of steps S101 to S102 in the above embodiment, which is not described herein again.
In the embodiment of the application, the server may obtain the first target task, determine to-be-processed data and a data processing mode indicated by the first target task according to the first target task, then obtain task fragment information from the database, determine a correspondence between a task identifier carried in the first target task and a fragment identifier according to the task fragment information, determine a fragment identifier corresponding to the task identifier carried in the first target task according to the correspondence between the task identifier and the fragment identifier, further, the server may obtain, according to the fragment identifier corresponding to the task identifier carried in the first target task, a fragment state of a target fragment corresponding to the fragment identifier of the task identifier carried in the first target task from the task fragment information, and perform data processing on the to-be-processed data indicated by the first target task in the target fragment according to the fragment state of the target fragment by using the data processing mode indicated by the first target task. It can be seen that whether to perform data processing is determined according to the fragment state of the fragment corresponding to the acquired task, so that data can be prevented from being repeatedly processed, the waiting time is set, the task can be acquired again when the waiting time is exceeded, the reduction of data processing efficiency caused by overlong waiting time can be avoided, the task can be prevented from being deadlocked, and the data processing efficiency can be improved.
Referring to fig. 3, fig. 3 is a schematic flowchart illustrating another data processing method according to an embodiment of the present disclosure. As shown in fig. 3, it is assumed that the task to be processed is task a, the number of servers in the server cluster is N, the number of fragments in the database is M, the number of subtasks corresponding to task a is also M, and the fragmentation state of the fragments in the database is controlled by the CAS lock mechanism. In specific implementation, when the task a is triggered, each server in the server cluster may randomly obtain a part of or all of the tasks in each subtask in the task a, and the server may determine how large the task amount is obtained according to the idle load amount, and when the idle load amount is large, the task amount that can be obtained for the task may be also large. After each server acquires the corresponding target task, each server needs to acquire the fragment corresponding to the target task in the database, so as to process the to-be-processed data indicated by the target task in the fragment.
The following description will take one server in a server cluster as an example. After the server acquires the target task, the server may first acquire a corresponding relationship between the task identifier and the fragment identifier from task fragment information in the database, to determine the fragment identifier corresponding to the target task according to the corresponding relationship and the task identifier carried in the target task, and then acquire a corresponding fragment lock according to the fragment identifier, where the fragment lock is a fragment lock of the fragment corresponding to the target task, and the fragment lock is latched in two states, one is in a "0" state, which indicates that the fragment is currently available, and the other is in a "1" state, which indicates that the fragment is currently unavailable, i.e., is occupied. If the fragmentation state of the fragmentation lock acquired by the server is a "0" state, that is, it indicates that the task is successful, the server may acquire the fragmentation corresponding to the fragmentation identifier from the database according to the fragmentation identifier, and the fragmentation state of the fragmentation lock may be updated from the "0" state to a "1" state to show that the current fragmentation is occupied, so that other servers may not acquire the fragmentation for data processing. After the server acquires the fragment corresponding to the target task, the data to be processed indicated by the target task can be processed in the fragment, and after the processing is completed, the fragment state of the fragment lock can be updated from a "1" state to a "0" state to show that the current fragment is not occupied, so that other servers can acquire the fragment to process the data. If the fragmentation state of the fragmentation lock acquired by the server is a "1" state, that is, it indicates that the task fails, the server may wait in a queue until the fragmentation state of the fragmentation lock is updated to a "0" state, and a waiting time may be set in order to ensure the efficiency of data processing. If the fragmentation state of the fragmentation lock acquired within the waiting time is updated to be a '0' state, subsequent data processing can be performed. Optionally, the fragmentation state of the fragmentation lock may be acquired according to a preset time interval. If the fragmentation state of the fragmentation lock acquired within the waiting time is not updated to be the '0' state, the waiting can be abandoned, namely the target task can be abandoned, the next task can be acquired again, and the task deadlock is avoided.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present disclosure. The data processing apparatus includes:
a first obtaining unit 401, configured to obtain a first target task, where the first target task belongs to a first to-be-processed sub-task, and the first to-be-processed sub-task is any one of a plurality of to-be-processed sub-tasks obtained by dividing a to-be-processed task;
a determining unit 402, configured to determine, according to the first target task, to-be-processed data and a data processing manner indicated by the first target task;
a second obtaining unit 403, configured to obtain task fragment information from a database, and obtain a fragment state of a target fragment corresponding to the first target task from the database according to the task fragment information;
a processing unit 404, configured to perform data processing on the to-be-processed data indicated by the first target task in the target segment by using the data processing manner indicated by the first target task according to the segment state of the target segment.
In an implementation manner, the first target task carries a task identifier, and the second obtaining unit 403 is specifically configured to:
acquiring the corresponding relation between the task identifier and the fragment identifier from the task fragment information;
determining a fragment identifier corresponding to the task identifier carried in the first target task according to the corresponding relationship between the task identifier and the fragment identifier;
and acquiring the fragmentation state of the target fragmentation corresponding to the fragmentation identifier of the task identifier carried in the first target task from the task fragmentation information according to the fragmentation identifier corresponding to the task identifier carried in the first target task.
In an implementation manner, the shard status includes a first status and a second status, where the first status is used to indicate that the target shard is not occupied by other servers in the server cluster, and the second status is used to indicate that the target shard is occupied by other servers in the server cluster, and the processing unit 404 is specifically configured to:
if the fragmentation state of the target fragmentation is the first state, acquiring a target fragmentation corresponding to a fragmentation identifier corresponding to a task identifier carried in the first target task from the database according to the fragmentation identifier corresponding to the task identifier carried in the first target task;
and performing data processing on the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
In an implementation manner, the processing unit 404 is specifically configured to:
if the fragmentation state of the target fragmentation is the second state, acquiring the fragmentation state of the target fragmentation from the task fragmentation information according to a preset time interval within a preset time length, and determining whether the fragmentation state of the target fragmentation is updated to the first state;
if the fragment state of the target fragment is determined to be updated to the first state, acquiring the target fragment corresponding to the fragment identifier corresponding to the task identifier carried in the first target task from the database according to the fragment identifier corresponding to the task identifier carried in the first target task;
and performing data processing on the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
In one implementation, the processing unit 404 is further configured to:
if the fragmentation state of the target fragmentation is determined not to be the first state, acquiring a second target task, wherein the second target task belongs to a second to-be-processed subtask, the second to-be-processed subtask is any one of the plurality of to-be-processed subtasks, and the first to-be-processed subtask is different from the second to-be-processed subtask;
and performing data processing on the data to be processed indicated by the second target task in the target fragment according to the data processing mode indicated by the second target task.
In one implementation, the processing unit 404 is further configured to:
updating the fragmentation state of the target fragmentation from the first state to the second state;
and updating the fragmentation state of the target fragment from the second state to the first state.
In an implementation manner, the first obtaining unit 401 is further configured to:
acquiring the task quantity of the first target task, and detecting whether the task quantity of the first target task is smaller than or equal to the idle load quantity of the server;
if the detection result is that the task quantity of the first target task is smaller than or equal to the idle load quantity of the server, executing the step of determining the to-be-processed data and the data processing mode indicated by the first target task according to the first target task;
and if the detection result shows that the task quantity of the first target task is larger than the idle load quantity of the server, pushing the first target task to other servers.
It can be understood that the functions of the functional units of the data processing apparatus described in the embodiment of the present application may be specifically implemented according to the method in the embodiment of the method described in fig. 1 or fig. 2, and the specific implementation process may refer to the description related to the embodiment of the method in fig. 1 or fig. 2, which is not described herein again.
In the embodiment of the present application, the first obtaining unit 401 obtains a first target task, where the first target task belongs to a first to-be-processed sub-task, the first to-be-processed sub-task is any one of a plurality of to-be-processed sub-tasks obtained by dividing the to-be-processed task, then, the determining unit 402 determines the data to be processed and the data processing manner indicated by the first target task according to the first target task, then, the second obtaining unit 403 obtains task fragment information from a database, and obtains a fragment state of a target fragment corresponding to the first target task from the database according to the task fragment information, and further, the processing unit 404 obtains, according to the fragment state of the target fragment, and performing data processing on the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task. By implementing the method, the concurrent data processing can be realized, the repeated data processing is avoided, and the data processing efficiency is improved.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a server according to an embodiment of the present disclosure. The server described in this embodiment is any one of a server cluster, and the server includes: a processor 501, a memory 502, and a network interface 503. The processor 501, the memory 502, and the network interface 503 may exchange data with each other.
The Processor 501 may be a Central Processing Unit (CPU), and may also be other general purpose processors, Digital Signal Processors (DSP), Application Specific Integrated Circuits (ASIC), Field-Programmable Gate arrays (FPGA) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and so on. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 502 may include both read-only memory and random access memory, and provides program instructions and data to the processor 501. A portion of the memory 502 may also include non-volatile random access memory. Wherein, the processor 501, when calling the program instruction, is configured to perform:
the method comprises the steps of obtaining a first target task, wherein the first target task belongs to a first to-be-processed subtask, and the first to-be-processed subtask is any one of a plurality of to-be-processed subtasks obtained by dividing the to-be-processed task;
determining to-be-processed data and a data processing mode indicated by the first target task according to the first target task;
acquiring task fragment information from a database, and acquiring a fragment state of a target fragment corresponding to the first target task from the database according to the task fragment information;
and according to the fragment state of the target fragment, performing data processing on the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
In an implementation manner, the first target task carries a task identifier, and the processor 501 is specifically configured to:
acquiring the corresponding relation between the task identifier and the fragment identifier from the task fragment information;
determining a fragment identifier corresponding to the task identifier carried in the first target task according to the corresponding relationship between the task identifier and the fragment identifier;
and acquiring the fragmentation state of the target fragmentation corresponding to the fragmentation identifier of the task identifier carried in the first target task from the task fragmentation information according to the fragmentation identifier corresponding to the task identifier carried in the first target task.
In an implementation manner, the fragmentation status includes a first status and a second status, where the first status is used to indicate that the target fragmentation is not occupied by other servers in the server cluster, and the second status is used to indicate that the target fragmentation is occupied by other servers in the server cluster, and the processor 501 is specifically configured to:
if the fragmentation state of the target fragmentation is the first state, acquiring a target fragmentation corresponding to a fragmentation identifier corresponding to a task identifier carried in the first target task from the database according to the fragmentation identifier corresponding to the task identifier carried in the first target task;
and performing data processing on the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
In one implementation, the processor 501 is specifically configured to:
if the fragmentation state of the target fragmentation is the second state, acquiring the fragmentation state of the target fragmentation from the task fragmentation information according to a preset time interval within a preset time length, and determining whether the fragmentation state of the target fragmentation is updated to the first state;
if the fragment state of the target fragment is determined to be updated to the first state, acquiring the target fragment corresponding to the fragment identifier corresponding to the task identifier carried in the first target task from the database according to the fragment identifier corresponding to the task identifier carried in the first target task;
and performing data processing on the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
In one implementation, the processor 501 is further configured to:
if the fragmentation state of the target fragmentation is determined not to be the first state, acquiring a second target task, wherein the second target task belongs to a second to-be-processed subtask, the second to-be-processed subtask is any one of the plurality of to-be-processed subtasks, and the first to-be-processed subtask is different from the second to-be-processed subtask;
and performing data processing on the data to be processed indicated by the second target task in the target fragment according to the data processing mode indicated by the second target task.
In one implementation, the processor 501 is further configured to:
updating the fragmentation state of the target fragmentation from the first state to the second state;
and updating the fragmentation state of the target fragment from the second state to the first state.
In one implementation, the processor 501 is further configured to:
acquiring the task quantity of the first target task, and detecting whether the task quantity of the first target task is smaller than or equal to the idle load quantity of the server;
if the detection result is that the task quantity of the first target task is smaller than or equal to the idle load quantity of the server, executing the step of determining the to-be-processed data and the data processing mode indicated by the first target task according to the first target task;
and if the detection result shows that the task quantity of the first target task is larger than the idle load quantity of the server, pushing the first target task to other servers.
In a specific implementation, the processor 501 and the memory 502 described in this embodiment of the present application may execute the implementation described in the data processing method provided in fig. 1 or fig. 2 in this embodiment of the present application, and may also execute the implementation of the data processing apparatus described in fig. 4 in this embodiment of the present application, which is not described herein again.
In this embodiment of the present application, the processor 501 may obtain a first target task, where the first target task belongs to a first to-be-processed subtask, where the first to-be-processed subtask is any one of a plurality of to-be-processed subtasks obtained by dividing a to-be-processed task, then determine, according to the first target task, to-be-processed data and a data processing mode indicated by the first target task, then obtain task fragmentation information from a database, and obtain, according to the task fragmentation information, a fragmentation state of a target fragmentation corresponding to the first target task from the database, and further, according to the fragmentation state of the target fragmentation, perform data processing on the to-be-processed data indicated by the first target task in the target fragmentation by using the data processing mode indicated by the first target task. By implementing the method, the concurrent data processing can be realized, the repeated data processing is avoided, and the data processing efficiency is improved.
The embodiment of the present application also provides a computer-readable storage medium, in which program instructions are stored, and when the program is executed, some or all of the steps of the data processing method in the embodiment corresponding to fig. 1 or fig. 2 may be included.
It should be noted that, for simplicity of description, the above-mentioned embodiments of the method are described as a series of acts or combinations, but those skilled in the art should understand that the present application is not limited by the order of acts described, as some steps may be performed in other orders or simultaneously according to the present application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by associated hardware instructed by a program, which may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.
It is emphasized that the data may also be stored in a node of a blockchain in order to further ensure the privacy and security of the data. The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
The foregoing detailed description is directed to a data processing method, an apparatus, a server, and a storage medium provided in the embodiments of the present application, and specific examples are applied in the present application to explain the principles and implementations of the present application, and the descriptions of the foregoing embodiments are only used to help understand the method and the core ideas of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
Claims (10)
1. A data processing method applied to a server, wherein the server is any one of a server cluster, and the method comprises:
the method comprises the steps of obtaining a first target task, wherein the first target task belongs to a first to-be-processed subtask, and the first to-be-processed subtask is any one of a plurality of to-be-processed subtasks obtained by dividing the to-be-processed task;
determining to-be-processed data and a data processing mode indicated by the first target task according to the first target task;
acquiring task fragment information from a database, and acquiring a fragment state of a target fragment corresponding to the first target task from the database according to the task fragment information;
and according to the fragment state of the target fragment, performing data processing on the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
2. The method of claim 1, wherein the first target task carries a task identity; the acquiring the fragment state of the target fragment corresponding to the first target task from the database according to the task fragment information includes:
acquiring the corresponding relation between the task identifier and the fragment identifier from the task fragment information;
determining a fragment identifier corresponding to the task identifier carried in the first target task according to the corresponding relationship between the task identifier and the fragment identifier;
and acquiring the fragmentation state of the target fragmentation corresponding to the fragmentation identifier of the task identifier carried in the first target task from the task fragmentation information according to the fragmentation identifier corresponding to the task identifier carried in the first target task.
3. The method of claim 2, wherein the shard status comprises a first status indicating that the target shard is not occupied by other servers in the server cluster and a second status indicating that the target shard is occupied by other servers in the server cluster; the performing, according to the fragmentation state of the target fragmentation, data processing on the to-be-processed data indicated by the first target task in the target fragmentation by using the data processing manner indicated by the first target task includes:
if the fragmentation state of the target fragmentation is the first state, acquiring a target fragmentation corresponding to a fragmentation identifier corresponding to a task identifier carried in the first target task from the database according to the fragmentation identifier corresponding to the task identifier carried in the first target task;
and performing data processing on the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
4. The method according to claim 3, wherein the performing, in the target segment, data processing on the to-be-processed data indicated by the first target task by using the data processing manner indicated by the first target task includes:
if the fragmentation state of the target fragmentation is the second state, acquiring the fragmentation state of the target fragmentation from the task fragmentation information according to a preset time interval within a preset time length, and determining whether the fragmentation state of the target fragmentation is updated to the first state;
if the fragment state of the target fragment is determined to be updated to the first state, acquiring the target fragment corresponding to the fragment identifier corresponding to the task identifier carried in the first target task from the database according to the fragment identifier corresponding to the task identifier carried in the first target task;
and performing data processing on the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task.
5. The method of claim 4, further comprising:
if the fragmentation state of the target fragmentation is determined not to be the first state, acquiring a second target task, wherein the second target task belongs to a second to-be-processed subtask, the second to-be-processed subtask is any one of the plurality of to-be-processed subtasks, and the first to-be-processed subtask is different from the second to-be-processed subtask;
and performing data processing on the data to be processed indicated by the second target task in the target fragment according to the data processing mode indicated by the second target task.
6. The method according to claim 3 or 4, wherein after obtaining the target segment corresponding to the segment identifier corresponding to the task identifier carried in the first target task from the database, the method further comprises:
updating the fragmentation state of the target fragmentation from the first state to the second state;
after the data processing is performed on the to-be-processed data indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task, the method further includes:
and updating the fragmentation state of the target fragment from the second state to the first state.
7. The method according to claim 1, wherein after the obtaining of the first target task and before the determining of the to-be-processed data and the data processing manner indicated by the first target task according to the first target task, further comprising:
acquiring the task quantity of the first target task, and detecting whether the task quantity of the first target task is smaller than or equal to the idle load quantity of the server;
if the detection result is that the task quantity of the first target task is smaller than or equal to the idle load quantity of the server, executing the step of determining the to-be-processed data and the data processing mode indicated by the first target task according to the first target task;
and if the detection result shows that the task quantity of the first target task is larger than the idle load quantity of the server, pushing the first target task to other servers.
8. A data processing apparatus, comprising:
the device comprises a first acquisition unit, a first processing unit and a second acquisition unit, wherein the first acquisition unit is used for acquiring a first target task, the first target task belongs to a first to-be-processed subtask, and the first to-be-processed subtask is any one of a plurality of to-be-processed subtasks obtained by dividing the to-be-processed task;
the determining unit is used for determining the data to be processed and the data processing mode indicated by the first target task according to the first target task;
the second acquisition unit is used for acquiring task fragment information from a database and acquiring the fragment state of a target fragment corresponding to the first target task from the database according to the task fragment information;
and the processing unit is used for performing data processing on the data to be processed indicated by the first target task in the target fragment by using the data processing mode indicated by the first target task according to the fragment state of the target fragment.
9. A server, comprising a processor, a memory, and a network interface, the processor, the memory, and the network interface being interconnected, wherein the memory is configured to store a computer program comprising program instructions, and wherein the processor is configured to invoke the program instructions to perform the method of any of claims 1-7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions that, when executed by a processor, cause the processor to carry out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011545402.1A CN112541041A (en) | 2020-12-23 | 2020-12-23 | Data processing method, device, server and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011545402.1A CN112541041A (en) | 2020-12-23 | 2020-12-23 | Data processing method, device, server and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112541041A true CN112541041A (en) | 2021-03-23 |
Family
ID=75017888
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011545402.1A Pending CN112541041A (en) | 2020-12-23 | 2020-12-23 | Data processing method, device, server and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112541041A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113515516A (en) * | 2021-08-09 | 2021-10-19 | 北京字节跳动网络技术有限公司 | Data processing method and device, computer equipment and computer storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110113387A (en) * | 2019-04-17 | 2019-08-09 | 深圳前海微众银行股份有限公司 | A kind of processing method based on distributed batch processing system, apparatus and system |
| CN110209496A (en) * | 2019-05-20 | 2019-09-06 | 中国平安财产保险股份有限公司 | Task sharding method, device and sliced service device based on data processing |
| CN110399394A (en) * | 2018-04-16 | 2019-11-01 | 北京京东尚科信息技术有限公司 | The data traversal method, apparatus and calculate node of calculate node |
| CN111752961A (en) * | 2020-06-29 | 2020-10-09 | 深圳前海微众银行股份有限公司 | Data processing method and device |
-
2020
- 2020-12-23 CN CN202011545402.1A patent/CN112541041A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110399394A (en) * | 2018-04-16 | 2019-11-01 | 北京京东尚科信息技术有限公司 | The data traversal method, apparatus and calculate node of calculate node |
| CN110113387A (en) * | 2019-04-17 | 2019-08-09 | 深圳前海微众银行股份有限公司 | A kind of processing method based on distributed batch processing system, apparatus and system |
| CN110209496A (en) * | 2019-05-20 | 2019-09-06 | 中国平安财产保险股份有限公司 | Task sharding method, device and sliced service device based on data processing |
| CN111752961A (en) * | 2020-06-29 | 2020-10-09 | 深圳前海微众银行股份有限公司 | Data processing method and device |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113515516A (en) * | 2021-08-09 | 2021-10-19 | 北京字节跳动网络技术有限公司 | Data processing method and device, computer equipment and computer storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107528882B (en) | Method and device for processing consensus request in block chain consensus network and electronic equipment | |
| US9304807B2 (en) | Fault tolerant batch processing | |
| CN108572876B (en) | Method and device for realizing read-write lock | |
| CN109191287B (en) | Block chain intelligent contract fragmentation method and device and electronic equipment | |
| CN110162344B (en) | Isolation current limiting method and device, computer equipment and readable storage medium | |
| CN108763963B (en) | Distributed processing method, device and system based on data access authority | |
| CN110069217B (en) | Data storage method and device | |
| CN110806942A (en) | Data processing method and device | |
| CN112541041A (en) | Data processing method, device, server and storage medium | |
| CN109542922B (en) | Processing method for real-time service data and related system | |
| CN110750517A (en) | Data processing method, device and equipment of local storage engine system | |
| CN113946427A (en) | Task processing method, processor and storage medium for multi-operating system | |
| CN112015563A (en) | Message queue switching method and device, electronic equipment and storage medium | |
| CN111752961A (en) | Data processing method and device | |
| CN111538785A (en) | Data writing method, device and system of block chain and electronic equipment | |
| CN111259012A (en) | Data homogenizing method and device, computer equipment and storage medium | |
| US10310916B2 (en) | Scalable spinlocks for non-uniform memory access | |
| CN106815061B (en) | Service processing method and device | |
| CN115934272A (en) | Online batch task processing method and device | |
| CN114047954A (en) | Data processing method and device based on register | |
| CN112799791A (en) | Method and device for calling distributed lock, electronic equipment and storage medium | |
| CN111639129A (en) | Transaction processing method and device, electronic equipment and computer-readable storage medium | |
| CN110839085A (en) | Instruction management method, instruction management device, electronic equipment and storage medium | |
| CN115361283B (en) | K8S-based cloud host management method and device and computer equipment | |
| CN115002106B (en) | Block chain verification method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |