CN115408128A

CN115408128A - Data processing method and device, electronic equipment and computer readable storage medium

Info

Publication number: CN115408128A
Application number: CN202211037813.9A
Authority: CN
Inventors: 贺志刚; 刘波; 赖安兵
Original assignee: Electronic Science And Technology Park Co ltd
Current assignee: Electronic Science And Technology Park Co ltd
Priority date: 2022-08-26
Filing date: 2022-08-26
Publication date: 2022-11-29

Abstract

The application provides a data processing method, a data processing device, an electronic device and a computer readable storage medium, wherein the method comprises the following steps: generating data to be processed by a data producer, wherein the data to be processed comprises a data key, processing links and a data total number, the data key is a first key value, and the processing links are used for identifying a processing sequence of the data to be processed; when the total data number is larger than one, storing the data to be processed into a data waiting queue until the data quantity taking the data key as a first key value in the data waiting queue reaches the total data number; and sequentially processing all the data of which the data key is the first key value according to the sequence marked by the processing links carried by the data of which the data key is the first key value. The method can improve the accuracy of data processing.

Description

Data processing method and device, electronic equipment and computer readable storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data processing method and apparatus, an electronic device, and a computer-readable storage medium.

Background

In the prior art, a data producer generates data and then pushes the data to a data queue, and then data are distributed to each data consumer by a data distribution service, so that the data are processed in a multi-linear parallel mode. Often in the order in which the data is received by the data consumer, may result in inconsistent results as expected from the data processing.

Disclosure of Invention

In view of this, embodiments of the present application provide a data processing method, an apparatus, an electronic device, and a computer-readable storage medium, so as to solve the problem of data processing in the prior art that an out-of-order problem exists.

In a first aspect, an embodiment of the present application provides a data processing method, including: generating data to be processed by a data producer, wherein the data to be processed comprises a data key, processing links and a data total number, the data key is a first key value, and the processing links are used for identifying the processing order of the data to be processed; when the total data number is more than one, storing the data to be processed into a data waiting queue until the data quantity taking the data key as a first key value in the data waiting queue reaches the total data number; and sequentially processing all the data of which the data key is the first key value according to the sequence marked by the processing links carried by the data of which the data key is the first key value.

In an optional implementation manner, the sequentially processing, according to the sequence identified by the processing links carried by the data key that is the first key value, all the data whose data key is the first key value includes: according to the processing of each item of data with the data key as the first key value, the data are successively distributed to data consumers; and sequentially processing each item of data with the data key as the first key value by the data consumer.

In the above embodiment, the data may be distributed to the data consumers in order, so that the data processing order of the data consumers is the same as the expected order, and the execution result can meet the requirement.

In an optional implementation manner, the sequentially processing all the data of which the data key is the first key value according to the sequence identified by the processing links carried by the data of which the data key is the first key value includes: transferring each item of data with a data key as the first key value from the data waiting queue to a data consumption queue, wherein the data consumption queue is used for storing the data waiting for processing; and sequentially acquiring and processing data from the data consumption queue by a data consumer according to the processing of each item of data taking a data key as the first key value.

In the above embodiment, the data consumption queue is introduced, and the data consumption queue and the data waiting queue are matched, so that the received data and the data which can be processed are respectively stored in the two queues, the data processing error can be reduced, and the effectiveness and the correctness of data processing are improved.

In an optional embodiment, the unloading, from the data waiting queue to a data consuming queue, each item of data of which the data key is the first key value, includes: and transferring each item of data with the data key in the data waiting queue as the first key value into a data consumption queue according to the sequence identified by the processing connection.

In the above embodiment, the data can be stored in the data consumption queue in sequence, so that the identification of the sequence can be reduced in the subsequent processing stage, the processing efficiency can be improved, and the accuracy of data processing can be improved.

In an optional embodiment, the storing the data to be processed into a data waiting queue until the amount of data in the data waiting queue with a data key as a first key value reaches the total number of data includes: storing the data to be processed into the data waiting queue; if the record information of the first key value exists at present, updating the current quantity of the data under the first key value; if the record information of the first key value does not exist currently, recording the first key value and the current quantity of data under the first key value; until the current quantity of the data under the first key value is equal to the total quantity of the data.

In the embodiment, the data stored in the data waiting queue can be recorded, whether the collection of each group of data is completed can be more clearly seen, the collected data can be processed in time, and the timeliness of data processing is improved.

In an alternative embodiment, the method further comprises: recording the initial time of the data to be processed corresponding to the first key value; and if the current quantity of the data under the first key value is less than the total quantity of the data and the time difference between the initial time and the current time exceeds a time threshold, deleting the data of which the data key in the data waiting queue is the first key value.

In the above embodiment, the time for collecting data may be recorded, and the data waiting queue may be occupied to avoid the data that is abnormal and cannot be collected completely.

In an optional embodiment, the method further comprises: and when the total number of the data is equal to one, storing the data to be processed into a data consumption queue to wait for a data consumer to acquire and process the data to be processed from the data consumption queue, wherein the data consumption queue is used for storing the data to be processed.

In the above embodiment, since there is no problem of processing order when only one item of data is included, data composed of one piece of data to be processed can be directly stored in the data consumption queue to wait for processing by the data consumer, which can reduce the processing efficiency of only one item of data.

In a second aspect, an embodiment of the present application further provides a data processing apparatus, including:

the data processing device comprises a generating module, a processing module and a processing module, wherein the generating module is used for generating data to be processed through a data producer, the data to be processed comprises a data key, processing links and a data total number, the data key is a first key value, and the processing links are used for identifying the processing sequence of the data to be processed;

the first storage module is used for storing the data to be processed into a data waiting queue when the total data number is more than one, until the data quantity taking the data key as the first key value in the data waiting queue reaches the total data number;

and the processing module is used for sequentially processing all the data of which the data key is the first key value according to the sequence marked by the processing links carried by the data of which the data key is the first key value.

In a third aspect, an embodiment of the present application further provides an electronic device, including: a processor, a memory storing machine-readable instructions executable by the processor, the machine-readable instructions being executable by the processor to perform the steps of the method described above when the electronic device is run.

In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to perform the steps of the above method.

The data processing method, the data processing device, the electronic equipment and the computer readable storage medium provided by the embodiment of the application are used for setting processing connection for data to identify a data processing sequence, and in addition, data which are not collected can be stored in a data waiting queue and processed until the data are collected, so that the correctness of data processing can be improved, and a processing result can be more expected.

In order to make the aforementioned objects, features and advantages of the present application comprehensible, embodiments accompanied with figures are described in detail below.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained from the drawings without inventive effort.

Fig. 1 is a block diagram of an electronic device according to an embodiment of the present disclosure;

fig. 2 is a flowchart of a data processing method according to an embodiment of the present application;

FIG. 3 is an alternative flowchart of step 230 of a data processing method provided by an embodiment of the present application;

FIG. 4 is another alternative flowchart of step 230 of the data processing method provided by the embodiment of the present application;

FIG. 5 is an alternative flowchart of step 220 of a data processing method provided in an embodiment of the present application;

FIG. 6 is a partial flowchart of a data processing method according to an embodiment of the present application;

fig. 7 is a schematic functional block diagram of a data processing apparatus according to an embodiment of the present disclosure.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.

The inventor researches and learns that in data processing, data are generated by a data producer and then processed by a data consumer, the data producer puts the generated data into a data queue, and then the data are distributed to each consumer (data processing service) by a data distribution service, so that the data are processed in a multi-linear parallel mode. Inevitably, however, there are some cases where data processing is sequential, for example, there are three pieces of data to be processed, which are: newly adding data, modifying data and deleting data, wherein the sequence is as follows: 1. data are added, data are modified, and data are deleted, so that due to concurrent multi-thread processing, the data processing sequence may be different from the expected sequence, and an exception occurs in the data processing result, for example: the deletion of data is executed first, the addition of data and the modification of data are executed, and the final execution result may have the condition that the data to be deleted is not deleted, so that data residue is caused and the data residue is inconsistent with the expected result.

Based on the current situation, a processing mode is provided, a data producer produces and adds a data processing Key (Key) and a processing time field, and data with the same data Key are distributed to the same memory queue by a data distribution service, so that the problem of execution consistency of a part of data is solved. However, this implementation method is still insufficient, and if data generated by a data producer arrives at the data queue in a delayed manner, the data put into the data queue may occupy the data queue for a long time, which causes a persistent abnormal data in the data queue, and thus the consistency of data execution may not be ensured.

Based on this, the data processing method, the data processing apparatus, the electronic device, and the computer-readable storage medium provided by the present application may execute data in groups, set an order for each item of data in a data group, and may be capable of executing in the set order when executed.

To facilitate understanding of the present embodiment, first, an electronic device executing the data processing method disclosed in the embodiments of the present application will be described in detail.

As shown in fig. 1, is a block schematic diagram of an electronic device. The electronic device 100 may include a memory 111, a processor 113. It will be understood by those of ordinary skill in the art that the structure shown in fig. 1 is merely exemplary and is not intended to limit the structure of the electronic device 100. For example, electronic device 100 may also include more or fewer components than shown in FIG. 1, or have a different configuration than shown in FIG. 1, depending on actual needs.

The memory 111 and the processor 113 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines. The processor 113 is used to execute the executable modules stored in the memory.

The Memory 111 may be, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Programmable Read-Only Memory (PROM), an Erasable Read-Only Memory (EPROM), an electrically Erasable Read-Only Memory (EEPROM), and the like. The memory 111 is used for storing a program, the processor 113 executes the program after receiving an execution instruction, and the method executed by the electronic device 100 defined by the process disclosed in any embodiment of the present application may be applied to the processor 113, or implemented by the processor 113.

The processor 113 may be an integrated circuit chip having signal processing capability. The Processor 113 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the Integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The electronic device 100 in this embodiment may be configured to perform each step in each method provided in this embodiment. The implementation of the data processing method is described in detail below by means of several embodiments.

Please refer to fig. 2, which is a flowchart illustrating a data processing method according to an embodiment of the present disclosure. The data processing method in the present embodiment may be executed by the electronic device shown in fig. 1. The specific process shown in fig. 2 will be described in detail below.

Step 210, generating data to be processed by a data producer.

The data to be processed comprises data keys, processing links and total data.

The data key of the data to be processed is a first key value, and the processing is used for identifying the processing order of the data to be processed.

Optionally, the data key may include a data class identifier and a data identifier. The data classification identifier can be used for representing the classification of the data to be processed, and the data identifier can be used for distinguishing the data to be processed.

And the data key of each piece of data to be processed is used for identifying the data group to which the data to be processed belongs. Each data group comprises at least one piece of data to be processed. The data keys of the data to be processed of any two data groups are different; the data keys of all the data to be processed in the same data group are the same.

The above process is used to indicate the number of bits to be processed. In one example, if a data group theoretically should include N pieces of data to be processed, the processing of each piece of data to be processed in the data group takes a positive integer less than or equal to N. Wherein N is a positive integer greater than or equal to 1. The smaller the value of the processing connection is, the more advanced the execution sequence of the data to be processed is.

Optionally, the data to be processed may further include a data generation time. The data generation time is used for recording the time when the data producer produces the data to be processed.

Step 220, when the total number of data carried by the data to be processed is greater than one, storing the data to be processed into a data waiting queue until the data volume in the data waiting queue, which takes the data key as the first key value, reaches the total number of the data.

For example, if the total number of data carried by the to-be-processed data of a data group is N, when the number of the to-be-processed data in the data group received by the data waiting queue is N, a subsequent processing flow may be started; when the number of the data waiting queue receiving the data to be processed in the data group is smaller than N, the data to be processed in the data group in the data queue is only stored in the data waiting queue, and other data to be processed of the data group is waited to be stored in the data waiting queue.

Step 230, sequentially processing all the data with the data key as the first key value according to the sequence marked by the processing links carried by the data with the data key as the first key value.

In one example, if a data group includes 3 pieces of data to be processed, the data to be processed a, the data to be processed B, and the data to be processed C are respectively. The number of processes carried by the data to be processed a is 2, the number of processes carried by the data to be processed B is 3, and the number of processes carried by the data to be processed C is 1. The data set may be processed in the order of processing the data to be processed C first, then processing the data to be processed a, and finally processing the data to be processed B.

Through the processing mode, a group of data can be processed after the completed data to be processed is received, and the processing carried by the data to be processed and the sequence marked by the processing can be referred during processing, so that the data processing result can better meet the processing expectation.

In one embodiment, as shown in fig. 3, the step 230 may be implemented as follows:

and 231, distributing the data to the data consumers in sequence according to the processing of each item of data with the data key as the first key value.

With the same data set, values according to processing links can be distributed to data consumers from small to large in sequence.

Step 232, the data consumer sequentially processes the data items with the data key as the first key value.

With respect to the data consumer, the processing may be in order of the order in which the data to be processed is received.

By controlling the data distribution sequence, the data consumer can execute the data according to the received sequence, and the probability of data execution errors is reduced.

In another embodiment, another queue of data consumption queues may be introduced, which may be used to store data that can be processed. As shown in fig. 4, the step 230 may be implemented as follows:

step 233, transferring each item of data with the data key as the first key value from the data waiting queue to the data consuming queue.

The data consumption queue is used for storing data waiting for processing.

Optionally, each item of data in which the data key in the data waiting queue is the first key value is transferred to the data consumption queue according to the sequence identified by the processing sequence of the item of data.

In an example, if there may be N data keys with the number to be processed as the first key value, the N data to be processed may be sequentially stored in the data consumption queue according to the sequence of processing, which is from 1 to N.

For example, the storing and fetching rule of the data consumption queue may be that storing is first performed, fetching is performed, and then the data to be processed with the data key as the first key value may be stored in the data consumption queue according to the sequence of processing even 1 to N.

When the data to be processed is fetched, the data can be fetched in the order of 1 to N in processing.

And step 234, acquiring and processing data from the data consumption queue in sequence by the data consumer according to the processing of each item of data with the data key as the first key value.

In the above steps, the data to be processed are stored in the data consumption queue in sequence in advance, so that the data consumer can execute the data to be processed acquired from the data consumption queue without paying attention to the data execution sequence any more, and the efficiency of data processing can be improved.

In order to better understand the collection situation of each piece of data to be processed in the data waiting queue, the data to be processed in the data waiting queue can also be recorded. As shown in fig. 5, step 220 may include the following steps.

Step 221, store the data to be processed into the data waiting queue.

Step 222, if there is no record information of the first key value, recording the first key value and the current amount of data under the first key value.

Illustratively, the recording information may be implemented by one recording variable. In this embodiment, a record variable may be configured for each set of data in the data waiting queue, and the record variable is used to store the number of each set of data in the data waiting queue.

Illustratively, one record variable may be set for each data key. The record variable may be associated with a data key, and the record variable may also be named with the data key.

When data to be processed in any data group is stored in the data waiting queue for the first time, a record variable can be constructed for the data to be processed, and the initial value of the record variable is given as 1.

In step 223, if the record information of the first key value already exists, the current amount of data in the first key value is updated.

For example, every time the data to be processed with the data key as the first key value is stored in the data waiting queue, the record variable corresponding to the first key value may be incremented by one.

Until the current quantity of the data under the first key value is equal to the total quantity of the data, the data to be processed with the data key as the first key value can be subjected to subsequent processing.

In order to avoid occupying the data waiting queue for a long time, a waiting time limit can be set for the data of the data waiting queue. In an implementation manner, as shown in fig. 6, the data processing method of the present embodiment may further include the following steps.

Step 240, recording the initial time of the to-be-processed data corresponding to the first key value.

The initial time may be determined according to a field carried by the to-be-processed data when the to-be-processed data is generated by the data producer.

In an example, if the data to be processed with the three data keys as the first key values is stored in the data waiting queue, and the data amount of the data to be processed in the data group corresponding to the first key value is theoretically greater than three, the initial time may be set as a time at which the data generation time is earliest in the three data to be processed.

In an example, if data to be processed with three data keys as first key values are stored in the data waiting queue, and a data amount of the data to be processed in a data group corresponding to the first key value is greater than three in theory, the initial time may be set as a data generation time of the data to be processed, which is stored in the data waiting queue earliest, of the three data to be processed.

In step 250, if the current quantity of the data under the first key value is less than the total quantity of the data, and the time difference between the initial time and the current time exceeds the time threshold, the data with the data key in the data waiting queue as the first key value is deleted.

Illustratively, the time threshold may be set as desired, for example, the time threshold may be a duration of 20 seconds, 30 seconds, 40 seconds, and so on. Of course, the time threshold may be different from the above example, and may be set to other time lengths.

In another implementation, the data processing method of this embodiment may further include: recording the first storage time of the data to be processed corresponding to the first key value stored in the data waiting queue for the first time; and if the current quantity of the data under the first key value is less than the total quantity of the data and the time difference between the first storage time and the current time exceeds a time threshold, deleting the data of which the data key in the data waiting queue is the first key value.

The first time to store may be a time to store the data under the first key value into the data waiting queue for the first time.

In an example, if data to be processed with three data keys as first key values are stored in the data waiting queue, and a data amount of the data to be processed in a data group corresponding to the first key value is greater than three in theory, the first storing time may be set as a storing time of the data to be processed, which is stored in the data waiting queue earliest, of the three data to be processed.

Optionally, the data that is not collected beyond the time threshold may also be stored in an error data queue.

An error log may be generated from data in the error data queue, which may record a variety of information. The information may include data keys that cannot be processed, data generation time, latency, and the like. The wait time may be a time difference between an initial time and a time when the error data queue is stored. The wait time may also be the time difference between the time of the first store and the time of the store of the error data queue.

In some cases, one piece of data to be processed is an independent task and does not need to be processed along with other pieces of data to be processed, and the method may further include: when the total number of the data is equal to one, the data to be processed is stored in a data consumption queue to wait for a data consumer to acquire and process the data to be processed from the data consumption queue.

The data consumption queue is used for storing data waiting for processing.

Through the steps, the data group comprising a plurality of items of data can be distinguished from single data, and the accuracy of data processing can be improved under the condition of improving the data processing efficiency, so that the processing result can better meet the expectation.

In order to make the implementation process of the embodiments of the present application clearer, the implementation process of data processing may be described by some examples:

in one example, the following table 1 provides several pieces of data to be processed:

No	data Key	Processing with two	Total number of pieces	Time of data generation
					1	Aoa_No0001	2	3	2022-06-08 20:01:01
2	Aoa_No0001	1	3	2022-06-08 20:01:02
					3	Aoa_No0001	3	3	2022-06-08 20:01:03
4	Bob_No0001_1	1	2	2022-06-08 20:01:04
					5	Coc_No0001_1	1	1	2022-06-08 20:01:05

TABLE 1

The data producer sequentially generates five pieces of data to be processed according to the sequence of the table 1: the data to be processed Aoa _ No0001, the data to be processed Bob _ No0001_1 and the data to be processed Coc _ No0001_1.

The data producer generates a first piece of data to be processed, the data Key is Aoa _ No0001, and the processing is carried out in succession: 2, total number of data: 3; and after the total number of the data of the first piece of data to be processed is determined to be more than 1, storing the first piece of data to be processed into a data waiting queue.

After the data is stored into the data waiting queue, if the first piece of data to be processed finds that the current data Key data is not registered, the recording information of the data Key can be generated in the memory, and then the other 2 pieces of data to be processed are waited.

The data producer generates a second piece of data to be processed, wherein the data Key is Aoa _ No0001, and the processing is carried out by the following steps: 1, total number of data: 3. and after the total number of the data of the second piece of data to be processed is determined to be more than 1, storing the second piece of data to be processed into a data waiting queue.

After the data is stored into the data waiting queue, the second piece of data to be processed is known to find that the current data Key data is registered, the data is registered but the arrival number is smaller than the total number of the data, and after the recorded data of the piece of data in the memory is updated, the other 1 piece of data is started to wait.

And the data producer generates a third piece of data to be processed, wherein the data Key is Aoa _ No0001, and the processing is carried out in succession: 3, total number of data: 3. and after the total number of the third piece of data to be processed is determined to be more than 1, storing the third piece of data to be processed into a data waiting queue.

After the data is stored in the data waiting queue, the third piece of data to be processed is known to find that the current data Key data is registered, and the data is registered but the arrival number of the data is equal to the total number of the data. Then the other two pieces of data to be processed can be obtained, and the data to be processed are stored in the data consumption queue after being sorted according to the processing and connection sequence of each piece of data to be processed, and the record information of the data Key in the memory is deleted. Then, the data consumer processes the three pieces of data to be processed under the data Key in the order of processing consecutive 1,2,3.

The data producer generates the fourth piece of data to be processed, the data Key is Bob _ No0001, and the processing is carried out: 1, total number of data: 2. and after the total number of the data of the fourth piece of data to be processed is determined to be more than 1, storing the fourth piece of data to be processed into a data waiting queue.

After the data is stored into the data waiting queue, if the fourth piece of data to be processed finds that the current data Key data is not registered, the recording information of the data Key can be generated in the memory, and then the fourth piece of data to be processed starts to wait for other 1 piece of data to be processed.

But the subsequent data of which the data Key is Bob _ No0001 does not arrive for more than 30 seconds, the fourth piece of data to be processed is distributed to the error data queue, and the record information of the data Key in the memory is deleted.

The data producer generates a fifth piece of data to be processed, wherein the data Key is Coc _ No0001, and the processing is carried out in succession: 1, total number of data: 1. and after the total number of the data of the fifth piece of data to be processed is determined to be equal to 1, storing the fifth piece of data to be processed into a data consumption queue. And the data consumer acquires the fifth piece of data to be processed for processing.

Through the implementation mode, the data group comprising a plurality of pieces of data to be processed can be processed in order, and the processing result can better meet the expectation. In addition, the time for storing the data into the queue can be monitored, so that long-time invalid waiting can be avoided, and the effectiveness of data processing is improved.

Based on the same application concept, a data processing apparatus corresponding to the data processing method is also provided in the embodiments of the present application, and since the principle of the apparatus in the embodiments of the present application for solving the problem is similar to that in the embodiments of the data processing method, the implementation of the apparatus in the embodiments of the present application may refer to the description in the embodiments of the method, and repeated details are not repeated.

Please refer to fig. 7, which is a schematic diagram of functional modules of a data processing apparatus according to an embodiment of the present application. Each module in the data processing apparatus in this embodiment is configured to execute each step in the above-described method embodiment. The data processing apparatus includes: a generation module 310, a first storage module 320 and a processing module 330; the contents of each module are as follows:

a generating module 310, configured to generate data to be processed by a data producer, where the data to be processed includes a data key, a processing link and a total number of data, where the data key is a first key value, and the processing link is used to identify a processing order of the data to be processed;

a first storing module 320, configured to store the to-be-processed data into a data waiting queue when the total number of the data is greater than one, until an amount of data in the data waiting queue, which is a first key value with a data key, reaches the total number of the data;

the processing module 330 is configured to sequentially process all the data with the data key as the first key value according to the sequence identified by the processing links carried by the data with the data key as the first key value.

In a possible implementation manner, the processing module 330 is configured to sequentially distribute data to data consumers according to processing of each item of data whose data key is the first key value; and processing each item of data with the data key as the first key value in sequence by the data consumer.

In a possible implementation manner, the processing module 330 is configured to dump each item of data whose data key is the first key value from the data waiting queue to a data consuming queue, where the data consuming queue is used to store data waiting for processing; and sequentially acquiring and processing the data from the data consumption queue by the data consumer according to the processing of each item of data taking the data key as the first key value.

In a possible implementation manner, the processing module 330 is further configured to forward the data items of which the data key in the data waiting queue is the first key value to the data consumption queue according to the sequence identified by the processing of the data items.

In a possible implementation manner, the first storage module 320 is configured to store the data to be processed into the data waiting queue; if the record information of the first key value exists at present, updating the current quantity of the data under the first key value; if the record information of the first key value does not exist currently, recording the first key value and the current quantity of data under the first key value; until the current amount of data under the first key value is equal to the total amount of data.

In a possible implementation manner, the data processing apparatus of this embodiment may further include:

the deleting module is used for recording the initial time of the data to be processed corresponding to the first key value; and if the current quantity of the data under the first key value is less than the total quantity of the data and the time difference between the initial time and the current time exceeds a time threshold, deleting the data of which the data key in the data waiting queue is the first key value.

In a possible implementation manner, the data processing apparatus of this embodiment may further include: and the second storage module is used for storing the data to be processed into a data consumption queue when the total number of the data is equal to one so as to wait for a data consumer to acquire and process the data to be processed from the data consumption queue, and the data consumption queue is used for storing the data to be processed.

In addition, an embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program performs the steps of the data processing method in the foregoing method embodiment.

The computer program product of the data processing method provided in the embodiment of the present application includes a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the steps of the data processing method in the foregoing method embodiment, which may be specifically referred to in the foregoing method embodiment, and are not described herein again.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative and, for example, the flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

In addition, functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

The functions may be stored in a computer-readable storage medium if they are implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes. It should be noted that, in this document, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrases "comprising 8230; \8230;" 8230; "does not exclude the presence of additional like elements in a process, method, article, or apparatus that comprises the element.

The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method of data processing, comprising:

generating data to be processed by a data producer, wherein the data to be processed comprises a data key, processing links and a data total number, the data key of the data to be processed is a first key value, and the processing links are used for identifying a processing order of the data to be processed;

when the total number of the data is more than one, storing the data to be processed into a data waiting queue until the data volume which is the first key value with the data key in the data waiting queue reaches the total number of the data;

and sequentially processing all the data of which the data key is the first key value according to the sequence marked by the processing links carried by the data of which the data key is the first key value.

2. The method of claim 1, wherein the sequentially processing all the data with a data key as the first key value according to the sequence identified by the processing link carried by the data with a data key as the first key value comprises:

according to the processing of each item of data with the data key as the first key value, the data are successively distributed to data consumers;

and sequentially processing each item of data with the data key as the first key value by the data consumer.

3. The method according to claim 1 or 2, wherein the sequentially processing all the data of which the data key is the first key value according to the sequence identified by the processing links carried by the data of which the data key is the first key value comprises:

transferring each item of data with a data key as the first key value from the data waiting queue to a data consumption queue, wherein the data consumption queue is used for storing the data waiting for processing;

and sequentially acquiring and processing the data from the data consumption queue by the data consumer according to the processing of each item of data with the data key as the first key value.

4. The method of claim 3, wherein the unloading items of data for which the data key is the first key value from the data wait queue to a data consumption queue comprises:

and transferring each item of data with the data key in the data waiting queue as the first key value into a data consumption queue according to the sequence identified by the processing connection.

5. The method of claim 1, wherein storing the pending data in a data waiting queue until an amount of data corresponding to a data key as a first key value in the data waiting queue reaches the total amount of data, comprises:

storing the data to be processed into the data waiting queue;

if the record information of the first key value exists at present, updating the current quantity of the data under the first key value;

if the record information of the first key value does not exist currently, recording the first key value and the current quantity of data under the first key value;

until the current quantity of the data under the first key value is equal to the total quantity of the data.

6. The method of claim 5, further comprising:

recording the initial time of the data to be processed corresponding to the first key value;

and if the current quantity of the data under the first key value is less than the total quantity of the data and the time difference between the initial time and the current time exceeds a time threshold, deleting the data of which the data key in the data waiting queue is the first key value.

7. The method of claim 1, further comprising:

and when the total number of the data is equal to one, storing the data to be processed into a data consumption queue to wait for a data consumer to acquire and process the data to be processed from the data consumption queue, wherein the data consumption queue is used for storing the data to be processed.

8. A data processing apparatus, characterized by comprising:

the data processing device comprises a generating module, a processing module and a processing module, wherein the generating module is used for generating data to be processed through a data producer, the data to be processed comprises a data key, processing links and a data total number, the data key of the data to be processed is a first key value, and the processing links are used for identifying the processing sequence of the data to be processed;

the first storage module is used for storing the data to be processed into a data waiting queue when the total number of the data is more than one, until the data volume which is the first key value with the data key in the data waiting queue reaches the total number of the data;

9. An electronic device, comprising: a processor, a memory storing machine-readable instructions executable by the processor, the machine-readable instructions when executed by the processor performing the steps of the method of any one of claims 1 to 7 when the electronic device is run.

10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, is adapted to carry out the steps of the method according to any one of the claims 1 to 7.