CN112181686A - Data processing method, device and system, electronic equipment and storage medium - Google Patents
Data processing method, device and system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112181686A CN112181686A CN202011039859.5A CN202011039859A CN112181686A CN 112181686 A CN112181686 A CN 112181686A CN 202011039859 A CN202011039859 A CN 202011039859A CN 112181686 A CN112181686 A CN 112181686A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- preprocessing operation
- preprocessing
- offset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Hardware Redundancy (AREA)
Abstract
The embodiment of the invention provides a data processing method, a data processing device, a data processing system, electronic equipment and a storage medium. The method comprises the following steps: the method comprises the steps of reading data to be processed in a preset data consumption queue, executing a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting an offset, and restoring a data state to a state before the data to be processed is read from the preset data consumption queue when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed. According to the embodiment of the invention, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails, the whole first preprocessing operation, the whole second preprocessing operation and the whole third preprocessing operation fail, so that the data consumption process meets atomicity, and the problems of data loss and data repeated consumption caused by using an automatic submission offset mode are solved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method, apparatus, system, electronic device, and storage medium.
Background
Currently, a producer device may send produced data to a Kafka message queue for storage of the data via the Kafka message queue. Additionally, the consumer device may retrieve data from the Kafka consumption queue for consumption. Among them, the Kafka message queue is a high-throughput distributed publish/subscribe message queue.
Additionally, the consumer device may be configured to: after the read data is successfully written, the offset (offset) corresponding to the data it reads is committed to the Kafka consumption queue, i.e., the offset is manually committed, to inform the Kafka consumption queue of the location where the consumer device has read the data.
The inventor finds in the course of implementing the present invention that the following may exist for a consumer device: an exception occurs after storing read data a (e.g., a consumer device goes down), but the offset to which data a corresponds is not committed. In this case, when the consumer device returns to normal, the data a is again read from the Kafka consumption queue and stored, which results in repeated consumption of the data a.
Disclosure of Invention
Embodiments of the present invention provide a data processing method, apparatus, system, electronic device and storage medium, so as to avoid data loss when a consumer device is abnormal. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a data processing method, where the method includes:
reading data to be processed in a preset data consumption queue;
executing a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting the offset;
when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, restoring the data state to the state before the data to be processed is read from the preset data consumption queue.
Optionally, the method further comprises:
when the first preprocessing operation, the second preprocessing operation and the third preprocessing operation are successfully executed, finishing the processing of the data to be processed based on the first preprocessing operation, finishing the storage of the data to be processed based on the second preprocessing operation, and finishing the reporting of the offset based on the third preprocessing operation.
Optionally, the executing a first preprocessing operation for preprocessing the to-be-processed data, a second preprocessing operation for pre-storing the to-be-processed data, and a third preprocessing operation for pre-reporting the offset includes:
allocating a first service resource for a first preprocessing operation for preprocessing the data to be processed, wherein the first service resource comprises a computing resource and a cache resource; executing the first preprocessing operation by using the first service resource to obtain a preprocessing result of the data to be processed;
allocating a second service resource for a second preprocessing operation for pre-storing the data to be processed, wherein the second service resource comprises a computing resource and a storage resource; executing the second preprocessing operation by using the second service resource to obtain a pre-stored result of the data to be processed;
allocating a third service resource for a third preprocessing operation for pre-reporting the offset, wherein the third service resource comprises a computing resource, an interface resource and a cache resource; and executing the third preprocessing operation by using the third service resource to obtain a pre-reported offset result of the data to be processed.
Optionally, when the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation are all successfully executed, the step of completing processing of the to-be-processed data based on the first preprocessing operation, completing storage of the to-be-processed data based on the second preprocessing operation, and completing reporting of the offset based on the third preprocessing operation includes:
and when the first preprocessing operation, the second preprocessing operation and the third preprocessing operation are successfully executed, finishing the processing of the data to be processed based on the preprocessing result, finishing the storage of the data to be processed based on the pre-storing result, and finishing the reporting of the offset based on the pre-reporting offset result.
In a second aspect, an embodiment of the present invention provides a data processing method, where the method includes:
receiving a read request for data to be processed;
sending the data to be processed to consumer equipment so that the consumer equipment reads the data to be processed and executes a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting an offset;
and receiving the offset sent by the consumer equipment, and updating the data storage state.
Optionally, the step of reading, by the consumer device, the to-be-processed data, and executing a first preprocessing operation for preprocessing the to-be-processed data, a second preprocessing operation for pre-storing the to-be-processed data, and a third preprocessing operation for pre-reporting the offset includes:
reading data to be processed in a preset data consumption queue;
allocating a first service resource for a first preprocessing operation for preprocessing the data to be processed, wherein the first service resource comprises a computing resource and a cache resource; executing the first preprocessing operation by using the first service resource to obtain a preprocessing result of the data to be processed;
allocating a second service resource for a second preprocessing operation for pre-storing the data to be processed, wherein the second service resource comprises a computing resource and a storage resource; executing the second preprocessing operation by using the second service resource to obtain a pre-stored result of the data to be processed;
allocating a third service resource for a third preprocessing operation for pre-reporting the offset, wherein the third service resource comprises a computing resource, an interface resource and a cache resource; and executing the third preprocessing operation by using the third service resource to obtain a pre-reported offset result of the data to be processed.
Optionally, the method further comprises:
when the first preprocessing operation, the second preprocessing operation and the third preprocessing operation are successfully executed, finishing the processing of the data to be processed based on the first preprocessing operation, finishing the storage of the data to be processed based on the second preprocessing operation, and finishing the reporting of the offset based on the third preprocessing operation.
Optionally, when the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation are all successfully executed, the step of completing processing of the to-be-processed data based on the first preprocessing operation, completing storage of the to-be-processed data based on the second preprocessing operation, and completing reporting of the offset based on the third preprocessing operation includes:
and when the first preprocessing operation, the second preprocessing operation and the third preprocessing operation are successfully executed, finishing the processing of the data to be processed based on the preprocessing result, finishing the storage of the data to be processed based on the pre-storing result, and finishing the reporting of the offset based on the pre-reporting offset result.
In a third aspect, an embodiment of the present invention provides a data processing apparatus, where the apparatus includes:
the reading unit is used for reading data to be processed in a preset data consumption queue;
an execution unit, configured to execute a first preprocessing operation for preprocessing the to-be-processed data, a second preprocessing operation for pre-storing the to-be-processed data, and a third preprocessing operation for pre-reporting an offset;
a rollback unit, configured to, when at least one of the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation fails to be executed, restore a data state to a state before the to-be-processed data is read from the preset data consumption queue.
Optionally, the apparatus further comprises:
and the processing unit is used for finishing the processing of the data to be processed based on the first preprocessing operation, finishing the storage of the data to be processed based on the second preprocessing operation and finishing the reporting of the offset based on the third preprocessing operation when the first preprocessing operation, the second preprocessing operation and the third preprocessing operation are successfully executed.
Optionally, the execution unit includes:
the first sub-execution unit is used for allocating a first service resource for a first preprocessing operation for preprocessing the data to be processed, wherein the first service resource comprises a computing resource and a cache resource; executing the first preprocessing operation by using the first service resource to obtain a preprocessing result of the data to be processed;
the second sub-execution unit is used for allocating a second service resource for a second preprocessing operation for pre-storing the data to be processed, wherein the second service resource comprises a computing resource and a storage resource; executing the second preprocessing operation by using the second service resource to obtain a pre-stored result of the data to be processed;
a second sub-execution unit, configured to allocate a third service resource to a third preprocessing operation for reporting an offset in advance, where the third service resource includes a computation resource, an interface resource, and a cache resource; and executing the third preprocessing operation by using the third service resource to obtain a pre-reported offset result of the data to be processed.
Optionally, the processing unit is specifically configured to:
and when the first preprocessing operation, the second preprocessing operation and the third preprocessing operation are successfully executed, finishing the processing of the data to be processed based on the preprocessing result, finishing the storage of the data to be processed based on the pre-storing result, and finishing the reporting of the offset based on the pre-reporting offset result.
In a fourth aspect, an embodiment of the present invention provides a data processing apparatus, where the apparatus includes:
a receiving unit configured to receive a read request for data to be processed;
a sending unit, configured to send the data to be processed to consumer equipment, so that the consumer equipment reads the data to be processed, and executes a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed, and a third preprocessing operation for pre-reporting an offset;
and the updating unit is used for receiving the offset sent by the consumer equipment and updating the data storage state.
In a fifth aspect, an embodiment of the present invention further provides a data processing system, where the data processing system includes: a producer device, a data consumption queue, and a consumer device;
the producer device is used for generating data and sending the generated data to the data consumption queue;
the data consumption queue is used for receiving the data sent by the producer equipment, storing the data, sending the data to be processed to the consumer equipment when receiving a read request aiming at the data to be processed sent by the consumer equipment, receiving an offset sent by the consumer equipment, and updating a data storage state;
the consumer device is configured to send a read request for data to be processed to the data consumption queue, read the data to be processed in the data consumption queue, execute a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed, and a third preprocessing operation for pre-reporting an offset, and restore a data state to a state before the data to be processed is read from the preset data consumption queue when at least one of the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation fails to be executed.
In a sixth aspect, an embodiment of the present invention further provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor and the communication interface complete communication between the memory and the processor through the communication bus;
a memory for storing a computer program;
a processor adapted to perform the method steps of the first aspect or the second aspect when executing a program stored in the memory.
In a seventh aspect, an embodiment of the present invention further provides a storage medium, where the storage medium is a computer-readable storage medium, and a computer program is stored in the computer-readable storage medium, where the computer program, when executed by a processor, implements the method steps in the first aspect or any of the second aspects.
In an eighth aspect, an embodiment of the present invention further provides a computer program product, which when run on an electronic device, causes the electronic device to perform: the method steps of any one of the first aspect, or the second aspect.
In the embodiment of the present invention, when at least one of the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation fails to be executed, the data state may be restored to the state before the data to be processed is read from the preset data consumption queue, that is, when at least one of the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation fails to be executed, the entire first preprocessing operation, the second preprocessing operation, and the third preprocessing operation all fail, so that the data consumption process satisfies atomicity, data loss caused by using an automatic offset submitting manner is avoided, and the problem of data repeat consumption caused by restoring to normal is avoided when the offset is not submitted but the data is stored in the consumer device by using the existing manual offset submitting. Of course, it is not necessary for any product or method of practicing the invention to achieve all of the above-described advantages at the same time.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart illustrating another data processing method according to an embodiment of the present invention;
fig. 3 is a schematic flow chart illustrating another data processing method according to an embodiment of the present invention;
fig. 4 is a schematic flow chart of a data processing method according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
FIG. 6 is a block diagram of another data processing apparatus according to an embodiment of the present invention;
FIG. 7 is a block diagram of a data processing system according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Currently, a producer device may send produced data to a Kafka message queue for buffering the data via the Kafka message queue. Additionally, the consumer device may retrieve data from the Kafka consumption queue for consumption. Among them, the Kafka message queue is a high-throughput distributed publish/subscribe message queue.
Additionally, the consumer device may be configured to: after the read data is successfully written, the offset (offset) corresponding to the data it reads is committed to the Kafka consumption queue, i.e., the offset is manually committed, to inform the Kafka consumption queue of the location where the consumer device has read the data.
The inventor finds in the course of implementing the present invention that the following may exist for a consumer device: an exception occurs after storing read data a (e.g., a consumer device goes down), but the offset to which data a corresponds is not committed. In this case, when the consumer device returns to normal, the data a is again read from the Kafka consumption queue and stored, which results in repeated consumption of the data a.
In order to solve the above technical problem, an embodiment of the present invention provides a data processing method, where the method may include:
reading data to be processed in a preset data consumption queue;
executing a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting the offset;
when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, restoring the data state to the state before the data to be processed is read from the preset data consumption queue.
In the embodiment of the invention, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, the data state can be restored to the state before the data to be processed is read from the preset data consumption queue, that is, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, the whole first preprocessing operation, the second preprocessing operation and the third preprocessing operation fail, so that the data consumption process meets atomicity, the data loss caused by an automatic offset submitting mode is avoided, the existing manual offset submitting mode is used, the abnormality occurs under the condition that the consumer equipment stores the data but does not submit the offset, and the problem of data repeated consumption caused after the normal operation is restored. It is understood that the method of the present embodiment may be applied to Kafka, which is exemplified in the present embodiment, but the present invention is not limited thereto.
The following describes a data processing method provided by an embodiment of the present invention in detail:
the data processing method provided by the embodiment of the invention can be applied to a Kafka cluster, and the Kafka cluster is a distributed message system. In the embodiment of the present invention, a data processing method applied to a Kafka cluster is described as an example. The Kafka cluster may include producer devices, consumer devices, and Kafka consumption queues, among others. Wherein. Consumer devices may include, but are not limited to, servers, computers, cell phones, and the like.
In practical applications, the consumer device may set the function of auto-commit offset, which refers to: immediately after the consumer device reads data from the Kafka consumption queue, the offset (offset) corresponding to the data it reads is submitted to the Kafka consumption queue. This manner of committing the offset is likely to occur as follows: after reading the data and before storing the data, the consumer device exceptions, resulting in the offset corresponding to the data being committed to the Kafka consumption queue but the data is not stored, and the consumer device will read the data from where the offset begins next time, resulting in the loss of the previously read data.
In order to avoid data loss caused by automatic offset submission when the consumer device sets the function of automatic offset submission, in the embodiment of the present invention, it may be agreed in advance that the consumer device uses a manual offset submission mode in the Kafka cluster to notify the Kafka consumption queue of the offset corresponding to data consumption completed by the consumer device.
As shown in fig. 1, fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention. The data processing method may be applied in a consumer device, and the method may comprise the steps of:
s101, reading data to be processed in a preset data consumption queue.
In the embodiment of the present invention, the preset data consumption queue may be a Kafka consumption queue in a Kafka cluster. It will be appreciated that the consumer device may read pending data from the Kafka consumption queue, and that there may be two consequences to reading pending data: read success and read failure. After the data to be processed is successfully read, the consumer device executes step S102.
In the embodiment of the present invention, the read data to be processed may include picture data, text data, and audio data, but is not limited thereto.
As a preferred implementation manner of the embodiment of the present invention, the embodiment of the present invention may place, by introducing a transaction mechanism, the read of the to-be-processed data, the operation related to the read to-be-processed data, and the operation of committing the offset into the same transaction, that is, place the consumption of the to-be-processed data by the consumer device into one transaction at a time for operation. In this way, it can be guaranteed that all operations in a transaction are either all completed successfully or are not all completed and do not stay in one of the operations in the middle.
The transaction mechanism refers to a series of operations executed as a logic unit, and when data updating occurs synchronously, data inconsistency can be prevented through the transaction mechanism. To become a logical unit of work, atomicity, consistency, isolation, and durability must be satisfied, which are existing concepts and not described in detail herein.
S102, executing a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting the offset.
In the embodiment of the present invention, after the to-be-processed data is successfully read from the Kafka consumption queue, a first preprocessing operation for preprocessing the to-be-processed data, a second preprocessing operation for pre-storing the to-be-processed data, and a third preprocessing operation for pre-reporting the offset may be performed. The execution results of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation can be success or failure, when the execution results show success, the preprocessing operation on the data to be processed is shown to be successful, at this time, only the preprocessing operation corresponding to the data to be processed, of which the execution results show success, is frozen, and the data processing corresponding to the preprocessing operations is not really completed.
As a preferred implementation manner of the embodiment of the present invention, an implementation process of executing a first preprocessing operation for preprocessing data to be processed, a second preprocessing operation for pre-storing the data to be processed, and a third preprocessing operation for pre-reporting an offset may include:
allocating a first service resource for a first preprocessing operation for preprocessing the data to be processed, and executing the first preprocessing operation by using the first service resource to obtain a preprocessing result of the data to be processed.
Wherein the first service resource may include: computing resources and caching resources. The first preprocessing operation for preprocessing the data to be processed may include: and creating, modifying, screening, updating and the like the read data to be processed. The preprocessing result of the data to be processed may be a result indicating that the preprocessing failed, and a result indicating that the preprocessing succeeded including a processing result obtained by preprocessing the data to be processed.
For example, the first preprocessing operation is to modify the data to be processed, specifically, the first service resource is allocated to the first preprocessing operation for preprocessing the data to be processed, the first preprocessing operation is executed by using the first service resource, and the result of preprocessing the data to be processed may be obtained as follows: and caching the modifier of the data to be processed by utilizing the cache resource allocated to the first preprocessing operation to obtain a preprocessing result of the data to be processed.
And allocating a second service resource for a second preprocessing operation for pre-storing the data to be processed, and executing the second preprocessing operation by using the second service resource to obtain a pre-storing result of the data to be processed.
Wherein the second service resource may include: the computing resource and the cache resource, and the pre-storing result of the data to be processed may be a result indicating that the pre-storing fails and a result indicating that the pre-storing succeeds.
For example, the computing resources and the cache resources may be allocated for the second preprocessing operation, and then the cache space required for pre-storing the data to be processed is calculated by using the computing resources allocated for the second preprocessing operation, and the data to be processed is cached by using the cache resources allocated for the second preprocessing operation.
And allocating a third service resource for a third preprocessing operation for the pre-reporting offset, and executing the third preprocessing operation by using the third service resource to obtain a pre-reporting offset result of the data to be processed.
Wherein the third service resource may include: the result of the pre-reporting offset of the data to be processed can be a result indicating that the reporting fails and a result indicating that the reporting is successful.
For example, the computing resource, the interface resource, and the buffer resource may be allocated for the third preprocessing operation, and then the network resource and the buffer space required for calculating the pre-reporting offset may be calculated by using the computing resource allocated for the third preprocessing operation, the pre-reporting offset may be buffered by using the buffer resource allocated for the third preprocessing operation, and the pre-reporting offset may be pre-reported by using the interface resource allocated for the third preprocessing operation.
S103, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, restoring the data state to the state before the data to be processed is read from the preset data consumption queue.
In the embodiment of the present invention, a transaction mechanism may be introduced to place the reading of the to-be-processed data, and execute a first preprocessing operation for preprocessing the to-be-processed data, a second preprocessing operation for pre-storing the to-be-processed data, and a third preprocessing operation for pre-reporting the offset in the same transaction, so as to implement consumption of the to-be-processed data, and further, when at least one of the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation fails to be executed, it indicates that the attempted consumption of the to-be-processed data fails, and at this time, the data state is restored to the state before the to-be-processed data is read from the preset data consumption queue, so that the problem of data re-consumption caused by the existing offset manual submission manner can be avoided.
As a preferred implementation manner of the embodiment of the present invention, when at least one of the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation fails to be executed, or when the consumer device is abnormal, the data state may be restored to a state before the data to be processed is read from the preset data consumption queue. The abnormal condition of the consumer device includes that the consumer device is down or a network is abnormal.
As a preferred implementation manner of the embodiment of the present invention, the transaction mechanism introduced by the embodiment of the present invention may be a TCC (Try to Confirm cancellation mechanism) type transaction mechanism. The TCC calls the service logic only one temporary operation, and calls its consuming side service to reserve subsequent cancellation right, that is, when the current data consumption fails, the first preprocessing operation for preprocessing the data to be processed, the second preprocessing operation for pre-storing the data to be processed, and the third preprocessing operation for pre-reporting the offset are cancelled.
In the embodiment of the invention, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, the data state can be restored to the state before the data to be processed is read from the preset data consumption queue, that is, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, the whole first preprocessing operation, the second preprocessing operation and the third preprocessing operation fail, so that the data consumption process meets atomicity, the data loss caused by using an automatic offset submitting mode is avoided, the existing manual offset submitting mode is used, an exception occurs under the condition that the consumer equipment stores the data but does not submit the offset, and the problem of data repeated consumption caused after the normal operation is restored is solved.
As shown in fig. 2, another data processing method according to an embodiment of the present invention may include the following steps:
s201, reading data to be processed in a preset data consumption queue.
S202, a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed, and a third preprocessing operation for pre-reporting the offset are performed.
S203, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, restoring the data state to the state before the data to be processed is read from the preset data consumption queue.
The implementation process of steps S201 to S203 may be the same as steps S101 to S103, and the embodiment of the present invention is not described herein again.
S204, when the first preprocessing operation, the second preprocessing operation and the third preprocessing operation are successfully executed, the data to be processed are processed based on the first preprocessing operation, the data to be processed are stored based on the second preprocessing operation, and the offset is reported based on the third preprocessing operation.
In the embodiment of the present invention, a transaction mechanism may be introduced to read data to be processed, and a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed, and a third preprocessing operation for pre-reporting an offset are performed in the same transaction, so as to implement consumption of the data to be processed, and further, when the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation are performed successfully, it indicates that the attempted consumption of the data to be processed is successful, at this time, processing of the data to be processed is completed based on the first preprocessing operation, storage of the data to be processed is completed based on the second preprocessing operation, and reporting of the offset is completed based on the third preprocessing operation, so as to complete consumption of the data to be processed this time.
As a preferred implementation manner of the embodiment of the present invention, when the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation are all successfully executed, the processing of the data to be processed is completed based on the preprocessing result, the storage of the data to be processed is completed based on the pre-storing result, and the reporting of the offset is completed based on the pre-reporting offset result.
In the embodiment of the present invention, when the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation are all successfully executed, the processing of the data to be processed is completed by using the preprocessing result obtained by the first preprocessing operation on the data to be processed, the storage of the data to be processed is completed by using the pre-stored result obtained by the second preprocessing operation on the data to be processed, and the reporting of the offset is completed by using the pre-reported offset result obtained by the third preprocessing operation on the data to be processed.
In the embodiment of the invention, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, the data state can be restored to the state before the data to be processed is read from the preset data consumption queue, that is, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, the whole first preprocessing operation, the second preprocessing operation and the third preprocessing operation fail, so that the data consumption process meets atomicity, the data loss caused by an automatic offset submitting mode is avoided, the existing manual offset submitting mode is used, the abnormality occurs under the condition that the consumer equipment stores the data but does not submit the offset, and the problem of data repeated consumption caused after the normal operation is restored.
As shown in fig. 3, an embodiment of the present invention provides another data processing method, where the data processing method may be applied to a Kafka cluster, and in particular, may be applied to a Kafka consumption queue in the Kafka cluster, and the method may include the following steps:
s301, a reading request for data to be processed is received.
In the embodiment of the present invention, the Kafka consumption queue in the Kafka cluster may receive a read request for the to-be-processed data, and may further send the to-be-processed data to the consumer device.
S302, sending the data to be processed to the consumer device, so that the consumer device reads the data to be processed, and executes a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed, and a third preprocessing operation for pre-reporting the offset.
In the embodiment of the present invention, the Kafka consumption queue in the Kafka cluster may send the data to be processed to the consumer device, so that the consumer device reads the data to be processed, and executes a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed, and a third preprocessing operation for pre-reporting the offset, thereby completing consumption of the data to be processed. Specifically, the operation of reading the data to be processed and consuming the data to be processed by the consumer device may refer to the above description, and the embodiment of the present invention is not described herein again.
And S303, receiving the offset sent by the consumer equipment, and updating the data storage state.
When the consumer device successfully consumes the data to be processed, the Kafka consumption queue in the Kafka cluster may update the data storage state in the Kafka consumption queue according to the received offset sent by the consumer device.
In the embodiment of the invention, a Kafka consumption queue in a Kafka cluster receives a reading request aiming at data to be processed and sends the data to be processed to consumer equipment, so that the consumer equipment reads the data to be processed and executes a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting an offset, thereby completing the consumption of the data to be processed, and the Kafka consumption queue further updates the data storage state in the Kafka consumption queue according to the received offset sent by the consumer equipment. The consumer device can restore the data state to the state before the data to be processed is read from the preset data consumption queue when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails, namely when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails, the whole first preprocessing operation, the second preprocessing operation and the third preprocessing operation fail, so that the data consumption process meets atomicity, data loss caused by an automatic offset submitting mode is avoided, the existing manual offset submitting mode is used, an exception occurs under the condition that the consumer device stores the data but does not submit the offset, and the problem of data repeated consumption caused after the normal data is restored.
As a preferred implementation manner of the embodiment of the present invention, an implementation process of a consumer device reading data to be processed and executing a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed, and a third preprocessing operation for pre-reporting an offset may include:
reading data to be processed in a preset data consumption queue;
allocating a first service resource for a first preprocessing operation for preprocessing data to be processed, wherein the first service resource comprises a computing resource and a cache resource; executing a first preprocessing operation by utilizing a first service resource to obtain a preprocessing result of the data to be processed;
allocating a second service resource for a second preprocessing operation for pre-storing data to be processed, wherein the second service resource comprises a computing resource and a storage resource; executing a second preprocessing operation by utilizing a second service resource to obtain a pre-stored result of the data to be processed;
allocating a third service resource for a third preprocessing operation for pre-reporting the offset, wherein the third service resource comprises a computing resource, an interface resource and a cache resource; and executing a third preprocessing operation by using a third service resource to obtain a pre-reported offset result of the data to be processed.
As a preferred implementation manner of the embodiment of the present invention, when the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation are all successfully executed, the consumer device may complete processing of the to-be-processed data based on the first preprocessing operation, complete storage of the to-be-processed data based on the second preprocessing operation, and complete reporting of the offset based on the third preprocessing operation.
As a preferred implementation manner of the embodiment of the present invention, when the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation are all successfully executed, the consumer device completes processing of the to-be-processed data based on the first preprocessing operation, completes storage of the to-be-processed data based on the second preprocessing operation, and completes an implementation process of reporting the offset based on the third preprocessing operation, which may include:
and when the first preprocessing operation, the second preprocessing operation and the third preprocessing operation are successfully executed, finishing the processing of the data to be processed based on the preprocessing result, finishing the storage of the data to be processed based on the pre-storing result, and finishing the reporting of the offset based on the pre-reporting offset result.
Exemplarily, as shown in fig. 4, fig. 4 is a schematic flow chart of a data processing method according to an embodiment of the present invention. In the embodiment of the invention, the Kafka cluster can be established depending on a zookeeper cluster, and the zookeeper is a distributed application program coordination service with distributed and open source codes. Wherein the producer device may comprise: producer device 1, producer device 2, producer device 3, etc. The consumer device may include: consumer device 1, consumer device 2, consumer device 3, etc. The Kafka consumption queue may include: the data to be consumed in the proxy servers 1, 2, 3, etc., i.e., the proxy servers, are stored in the Kafka consumption queue. Wherein, the consumer device, the producer device and the proxy server all establish communication connection with the zookeeper.
Any one producer device may send the produced data to the proxy server for storage via the Kafka consumption queue. Additionally, the consumer device may retrieve data from the Kafka consumption queue for consumption. After reading the data, the consumer device may complete consumption of the data based on the transaction mechanism and the manner in which the offset is manually committed.
In particular, the consumer device may commit the offset by introducing a transaction mechanism and manually committing the offset after the consumer device stores the data as one operation in the transaction mechanism. Therefore, when the transaction execution fails, the state of the consumer equipment can be recovered, and the problem of repeated consumption of data caused by a manual submission offset mode can be solved.
It will be appreciated that the transaction mechanism may be generated according to the specific business logic to be implemented by the consumer device, and that the transaction mechanism may differ for different consumer devices.
For example, the operation corresponding to the transaction mechanism of the consumer device a consuming data generation may include: the method comprises the steps of reading data to be processed in a Kafka consumption queue, executing a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting an offset, and restoring a data state to a state before the data to be processed is read from the preset data consumption queue when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed.
Corresponding to the embodiment of the data processing method shown in fig. 1, the embodiment of the present invention further provides a corresponding apparatus embodiment. As shown in fig. 5, an embodiment of the present invention provides a data processing apparatus, which may include:
the reading unit 501 is configured to read data to be processed in a preset data consumption queue.
An executing unit 502 is configured to execute a first preprocessing operation for preprocessing the to-be-processed data, a second preprocessing operation for pre-storing the to-be-processed data, and a third preprocessing operation for pre-reporting the offset.
The rollback unit 503 is configured to, when at least one of the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation fails to be executed, restore the data state to a state before the to-be-processed data is read from the predetermined data consumption queue.
In the embodiment of the invention, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, the consumer device can restore the data state to the state before the data to be processed is read from the preset data consumption queue, that is, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, the whole first preprocessing operation, the second preprocessing operation and the third preprocessing operation fail, so that the data consumption process meets atomicity, the data loss caused by an automatic offset submitting mode is avoided, the existing manual offset submitting mode is used, the abnormality occurs under the condition that the consumer device stores the data but does not submit the offset, and the problem of repeated consumption of the data caused after the normal operation is restored.
Optionally, the apparatus further comprises:
and a processing unit (not shown in the figure), configured to, when the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation are all successfully executed, complete processing of the to-be-processed data based on the first preprocessing operation, complete storage of the to-be-processed data based on the second preprocessing operation, and complete reporting of the offset based on the third preprocessing operation.
Optionally, the execution unit 502 includes:
the first sub-execution unit is used for allocating a first service resource for a first preprocessing operation for preprocessing data to be processed, wherein the first service resource comprises a computing resource and a cache resource; and executing a first preprocessing operation by using the first service resource to obtain a preprocessing result of the data to be processed.
The second sub-execution unit is used for allocating second service resources for a second preprocessing operation for pre-storing data to be processed, wherein the second service resources comprise computing resources and storage resources; and executing a second preprocessing operation by utilizing a second service resource to obtain a pre-stored result of the data to be processed.
The second sub-execution unit is used for allocating a third service resource for a third preprocessing operation for pre-reporting the offset, wherein the third service resource comprises a computing resource, an interface resource and a cache resource; and executing a third preprocessing operation by using a third service resource to obtain a pre-reported offset result of the data to be processed.
Optionally, the processing unit is specifically configured to:
and when the first preprocessing operation, the second preprocessing operation and the third preprocessing operation are successfully executed, finishing the processing of the data to be processed based on the preprocessing result, finishing the storage of the data to be processed based on the pre-storing result, and finishing the reporting of the offset based on the pre-reporting offset result.
Corresponding to the embodiment of the data processing method shown in fig. 3, the embodiment of the present invention further provides a corresponding apparatus embodiment. As shown in fig. 6, an embodiment of the present invention provides a data processing apparatus, which may include:
the receiving unit 601 is configured to receive a read request for data to be processed.
A sending unit 602, configured to send the data to be processed to the consumer device, so that the consumer device reads the data to be processed, and executes a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed, and a third preprocessing operation for pre-reporting the offset.
The updating unit 603 is configured to receive the offset sent by the consumer device, and update the data storage status.
In the embodiment of the present invention, the data consumption queue may receive a read request for data to be processed, and send the data to be processed to the consumer device, so that the consumer device reads the data to be processed, and executes a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed, and a third preprocessing operation for pre-reporting an offset, thereby completing consumption of the data to be processed, and the data consumption queue further updates a data storage state in the data consumption queue according to the received offset sent by the consumer device. The consumer device can restore the data state to the state before the data to be processed is read from the preset data consumption queue when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails, namely when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails, the whole first preprocessing operation, the second preprocessing operation and the third preprocessing operation fail, so that the data consumption process meets atomicity, data loss caused by an automatic offset submitting mode is avoided, the existing manual offset submitting mode is used, an exception occurs under the condition that the consumer device stores the data but does not submit the offset, and the problem of data repeated consumption caused after the normal data is restored.
Corresponding to the method embodiment, the embodiment of the invention also provides a corresponding system embodiment. As shown in fig. 7, an embodiment of the present invention provides a data processing system, including: producer device 701, data consumption queue 702, and consumer device 703;
a producer device 701 for generating data and transmitting the generated data to a data consumption queue 702;
the data consumption queue 702 is configured to receive data sent by the producer device 701, store the data, send the data to be processed to the consumer device 703 when receiving a read request for the data to be processed sent by the consumer device 703, receive an offset sent by the consumer device 703, and update a data storage state;
the consumer device 703 is configured to send a read request for the to-be-processed data to the data consumption queue 702, read the to-be-processed data in the data consumption queue 702, execute a first preprocessing operation for preprocessing the to-be-processed data, a second preprocessing operation for pre-storing the to-be-processed data, and a third preprocessing operation for pre-reporting the offset, and restore the data state to a state before the to-be-processed data is read from the preset data consumption queue when at least one of the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation fails to be executed.
In the embodiment of the invention, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, the consumer device can restore the data state to the state before the data to be processed is read from the preset data consumption queue, that is, when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, the whole first preprocessing operation, the second preprocessing operation and the third preprocessing operation fail, so that the data consumption process meets atomicity, the data loss caused by an automatic offset submitting mode is avoided, the existing manual offset submitting mode is used, the abnormality occurs under the condition that the consumer device stores the data but does not submit the offset, and the problem of repeated consumption of the data caused after the normal operation is restored.
Corresponding to the method embodiment, the embodiment of the invention also provides electronic equipment. Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present invention. Referring to fig. 8, the system comprises a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802 and the memory 803 communicate with each other via the communication bus 804,
a memory 803 for storing a computer program;
the processor 801 is configured to implement the following steps when executing the program stored in the memory 803:
reading data to be processed in a preset data consumption queue;
executing a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting the offset;
when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, restoring the data state to the state before the data to be processed is read from the preset data consumption queue.
Or implementing the following steps:
receiving a read request for data to be processed;
sending the data to be processed to consumer equipment so that the consumer equipment reads the data to be processed and executes a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting the offset;
and receiving the offset sent by the consumer equipment, and updating the data storage state.
Corresponding to the foregoing method embodiment, an embodiment of the present invention further provides a readable storage medium, where the readable storage medium is a computer-readable storage medium, and a computer program is stored in the computer-readable storage medium, where the computer program, when executed by a processor, implements the following steps:
reading data to be processed in a preset data consumption queue;
executing a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting the offset;
when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, restoring the data state to the state before the data to be processed is read from the preset data consumption queue.
Or implementing the following steps:
receiving a read request for data to be processed;
sending the data to be processed to consumer equipment so that the consumer equipment reads the data to be processed and executes a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting the offset;
and receiving the offset sent by the consumer equipment, and updating the data storage state.
Corresponding to the above method embodiment, an embodiment of the present invention further provides a computer program product, which when run on an electronic device, causes the electronic device to perform the following steps:
reading data to be processed in a preset data consumption queue;
executing a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting the offset;
when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, restoring the data state to the state before the data to be processed is read from the preset data consumption queue.
Or performing the following steps:
receiving a read request for data to be processed;
sending the data to be processed to consumer equipment so that the consumer equipment reads the data to be processed and executes a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting the offset;
and receiving the offset sent by the consumer equipment, and updating the data storage state.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus, system, electronic device, storage medium, and computer program product embodiments, because they are substantially similar to the method embodiments, the description is relatively simple, and for related matters, reference may be made to some descriptions of the method embodiments.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (10)
1. A method of data processing, the method comprising:
reading data to be processed in a preset data consumption queue;
executing a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting the offset;
when at least one of the first preprocessing operation, the second preprocessing operation and the third preprocessing operation fails to be executed, restoring the data state to the state before the data to be processed is read from the preset data consumption queue.
2. The method of claim 1, further comprising:
when the first preprocessing operation, the second preprocessing operation and the third preprocessing operation are successfully executed, finishing the processing of the data to be processed based on the first preprocessing operation, finishing the storage of the data to be processed based on the second preprocessing operation, and finishing the reporting of the offset based on the third preprocessing operation.
3. The method of claim 2, wherein the performing the first preprocessing operation for preprocessing the data to be processed, the second preprocessing operation for pre-storing the data to be processed, and the third preprocessing operation for pre-reporting the offset comprises:
allocating a first service resource for a first preprocessing operation for preprocessing the data to be processed, wherein the first service resource comprises a computing resource and a cache resource; executing the first preprocessing operation by using the first service resource to obtain a preprocessing result of the data to be processed;
allocating a second service resource for a second preprocessing operation for pre-storing the data to be processed, wherein the second service resource comprises a computing resource and a storage resource; executing the second preprocessing operation by using the second service resource to obtain a pre-stored result of the data to be processed;
allocating a third service resource for a third preprocessing operation for pre-reporting the offset, wherein the third service resource comprises a computing resource, an interface resource and a cache resource; and executing the third preprocessing operation by using the third service resource to obtain a pre-reported offset result of the data to be processed.
4. The method of claim 3, wherein when the first pre-processing operation, the second pre-processing operation, and the third pre-processing operation are all successfully executed, the steps of completing processing of the to-be-processed data based on the first pre-processing operation, completing storing of the to-be-processed data based on the second pre-processing operation, and completing reporting of an offset based on the third pre-processing operation comprise:
and when the first preprocessing operation, the second preprocessing operation and the third preprocessing operation are successfully executed, finishing the processing of the data to be processed based on the preprocessing result, finishing the storage of the data to be processed based on the pre-storing result, and finishing the reporting of the offset based on the pre-reporting offset result.
5. A method of data processing, the method comprising:
receiving a read request for data to be processed;
sending the data to be processed to consumer equipment so that the consumer equipment reads the data to be processed and executes a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed and a third preprocessing operation for pre-reporting an offset;
and receiving the offset sent by the consumer equipment, and updating the data storage state.
6. A data processing apparatus, characterized in that the apparatus comprises:
the reading unit is used for reading data to be processed in a preset data consumption queue;
an execution unit, configured to execute a first preprocessing operation for preprocessing the to-be-processed data, a second preprocessing operation for pre-storing the to-be-processed data, and a third preprocessing operation for pre-reporting an offset;
a rollback unit, configured to, when at least one of the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation fails to be executed, restore a data state to a state before the to-be-processed data is read from the preset data consumption queue.
7. A data processing apparatus, characterized in that the apparatus comprises:
a receiving unit configured to receive a read request for data to be processed;
a sending unit, configured to send the data to be processed to consumer equipment, so that the consumer equipment reads the data to be processed, and executes a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed, and a third preprocessing operation for pre-reporting an offset;
and the updating unit is used for receiving the offset sent by the consumer equipment and updating the data storage state.
8. A data processing system, characterized in that the data processing system comprises: a producer device, a data consumption queue, and a consumer device;
the producer device is used for generating data and sending the generated data to the data consumption queue;
the data consumption queue is used for receiving the data sent by the producer equipment, storing the data, sending the data to be processed to the consumer equipment when receiving a read request aiming at the data to be processed sent by the consumer equipment, receiving an offset sent by the consumer equipment, and updating a data storage state;
the consumer device is configured to send a read request for data to be processed to the data consumption queue, read the data to be processed in the data consumption queue, execute a first preprocessing operation for preprocessing the data to be processed, a second preprocessing operation for pre-storing the data to be processed, and a third preprocessing operation for pre-reporting an offset, and restore a data state to a state before the data to be processed is read from the preset data consumption queue when at least one of the first preprocessing operation, the second preprocessing operation, and the third preprocessing operation fails to be executed.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 4 or the method steps of claim 5 when executing a program stored in the memory.
10. A storage medium, characterized in that the storage medium is a computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method steps of any one of claims 1 to 4, or carries out the method steps of claim 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011039859.5A CN112181686A (en) | 2020-09-28 | 2020-09-28 | Data processing method, device and system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011039859.5A CN112181686A (en) | 2020-09-28 | 2020-09-28 | Data processing method, device and system, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112181686A true CN112181686A (en) | 2021-01-05 |
Family
ID=73944653
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011039859.5A Pending CN112181686A (en) | 2020-09-28 | 2020-09-28 | Data processing method, device and system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112181686A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115658347A (en) * | 2022-11-04 | 2023-01-31 | 拉卡拉支付股份有限公司 | Data consumption method, device, electronic equipment, storage medium and program product |
-
2020
- 2020-09-28 CN CN202011039859.5A patent/CN112181686A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115658347A (en) * | 2022-11-04 | 2023-01-31 | 拉卡拉支付股份有限公司 | Data consumption method, device, electronic equipment, storage medium and program product |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111666162B (en) | Distributed message transmission method, device, computer equipment and storage medium | |
| CN107016029B (en) | Method, device and system for processing service data | |
| EP3869434A1 (en) | Blockchain-based data processing method and apparatus, device, and medium | |
| CN112148798A (en) | Data processing method and device applied to distributed system | |
| US20200167150A1 (en) | Upgrading an execution environment for event-driven functions | |
| CN112667382A (en) | Task scheduling method, device, equipment and storage medium | |
| CN111090532A (en) | Application service calling method and device, electronic equipment and computer storage medium | |
| US10599629B2 (en) | Reduced locking in a distributed data store | |
| CN113032176A (en) | Distributed transaction double-compensation method and device based on daily account checking | |
| WO2020258653A1 (en) | Cross-node data processing method and apparatus | |
| CN115544044A (en) | Data consistency keeping method, device, equipment and storage medium | |
| CN115220876A (en) | Virtual resource creating method, device, program product, medium and electronic equipment | |
| CN110941622A (en) | Data processing method and device | |
| CN112181686A (en) | Data processing method, device and system, electronic equipment and storage medium | |
| CN114625566A (en) | Data disaster tolerance method and device, electronic equipment and storage medium | |
| CN110381150B (en) | Data processing method and device on block chain, electronic equipment and storage medium | |
| US9965538B2 (en) | Early thread return with secondary event writes | |
| CN114697398B (en) | Data processing method, device, electronic equipment, storage medium and product | |
| US20220276901A1 (en) | Batch processing management | |
| CN114116676A (en) | Data migration method and device, electronic equipment and computer readable storage medium | |
| CN110764933B (en) | Message processing method, device and system and computing equipment | |
| CN114036164A (en) | High-concurrency transaction locking method and system combining optimistic lock and pessimistic lock | |
| CN110489208B (en) | Virtual machine configuration parameter checking method, system, computer equipment and storage medium | |
| US11809910B2 (en) | System and method for dynamically resizing computational infrastructure to accommodate unexpected demands | |
| CN114124680B (en) | File access control alarm log management method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |