CN104199790A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN104199790A CN104199790A CN201410415527.0A CN201410415527A CN104199790A CN 104199790 A CN104199790 A CN 104199790A CN 201410415527 A CN201410415527 A CN 201410415527A CN 104199790 A CN104199790 A CN 104199790A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- queue
- level
- level cache
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 15
- 238000004519 manufacturing process Methods 0.000 claims abstract description 29
- 238000000034 method Methods 0.000 claims description 28
- 239000000284 extract Substances 0.000 claims description 27
- 238000013075 data extraction Methods 0.000 claims description 6
- 230000000903 blocking effect Effects 0.000 description 9
- 230000008569 process Effects 0.000 description 7
- 230000009471 action Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
Landscapes
- Memory System Of A Hierarchy Structure (AREA)
Abstract
The embodiment of the invention discloses a data processing method and device. The data processing method comprises the steps that a data production module produces at least one datum to be processed; the priorities are set for the data to be processed; the data to be processed are written in one-level buffer queues, corresponding to the priorities, of preset one-level buffer queue clusters according to the priorities of the data to be processed; according to the queue polling mode, a preset number of data which are to be processed and correspond to corresponding one-level buffer queues are extracted from the one-level buffer queues in all the one-level buffer queue clusters till all the data to be processed are extracted; the to-be-processed data extracted through polling every time are written in a preset two-level buffer queue in sequence. Obviously, by means of the scheme, the data with the high propriety can be processed as fast as possible rather than be blocked in the first-in first-out queues to wait for the data in front to be processed.
Description
Technical Field
The present invention relates to data processing methods, and in particular, to a data processing method and apparatus.
Background
During the development of system software, the following scenarios are often encountered: one module is responsible for generating data and another module is responsible for processing data. Wherein the module that generates the data is commonly referred to as a data production module (or producer); and the modules that process the data are often referred to as data processing modules (or consumers). In practical applications, the types of the data production module and the data processing module may be specifically a business system, a class, a function, a thread, a process, or the like.
In order to balance the data production activities of the data production module and the data processing activities of the data processing module, it is most critical to have a buffer queue between the data producer and the data processor. Specifically, the data production module writes the generated data to be processed into the buffer queue, and the data processing module extracts the data from the buffer queue for processing. The buffer queue may be a single congestion queue or a circular buffer queue, and the buffer queue is usually in the form of a FIFO first-in first-out queue.
Where processing data has priority requirements, the data in the buffer is typically reordered on a priority basis so that data of high priority is processed in preference to data of low priority. However, in some scenarios, the requirement on the priority is not strict, and it is not necessary to force the high-priority data to be processed before the low-priority data, but rather the high-priority data is required to be processed as soon as possible rather than blocking the first-in first-out queue waiting for the previous data to be processed.
Disclosure of Invention
Based on the above problems, embodiments of the present invention disclose a data processing method and apparatus to ensure that high priority data can be processed as soon as possible without blocking the queue of first in first out and waiting for the previous data to be processed. The technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a data processing method applied to a data processing system, where the data processing system includes a data production module for producing data and a data processing module for processing data; the method comprises the following steps:
the data production module produces at least one piece of data to be processed;
setting a priority for the at least one piece of data to be processed;
writing the at least one piece of data to be processed into a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed; the first-level cache queue set is used for placing data to be processed with the same priority;
sequentially extracting a preset number of data to be processed corresponding to a corresponding primary cache queue from the primary cache queues in each primary cache queue set according to a queue polling mode until at least one data to be processed is extracted;
and sequentially writing the data to be processed extracted by each polling into a preset second-level cache queue so that the data to be processed is extracted from the second-level cache queue by the data processing module for processing.
Optionally, the first-level cache queue is a single congestion queue;
the writing of the at least one piece of data to be processed into a first-level buffer queue in a preset first-level buffer queue set corresponding to the priority according to the priority of the at least one piece of data to be processed includes:
writing the at least one piece of data to be processed into the tail part of a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed;
according to the queue polling mode, sequentially extracting the data to be processed with the preset quantity corresponding to the corresponding first-level cache queue from the first-level cache queues in each first-level cache queue set, and the method comprises the following steps:
and sequentially extracting a preset amount of data to be processed corresponding to the corresponding first-level cache queue from the head of the first-level cache queue in each first-level cache queue set according to a queue polling mode.
Optionally, the first-level buffer queue is an annular buffer queue;
the writing of the at least one piece of data to be processed into a first-level buffer queue in a preset first-level buffer queue set corresponding to the priority according to the priority of the at least one piece of data to be processed includes:
writing the at least one piece of data to be processed into the head of a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed;
according to the queue polling mode, sequentially extracting the data to be processed with the preset quantity corresponding to the corresponding first-level cache queue from the first-level cache queues in each first-level cache queue set, and the method comprises the following steps:
and sequentially extracting a preset amount of data to be processed corresponding to the corresponding first-level cache queue from the tail part of the first-level cache queue in each first-level cache queue set according to a queue polling mode.
Optionally, the second-level cache queue is a single congestion queue;
the writing of the data to be processed extracted by each polling into a preset second-level cache queue in sequence comprises the following steps:
and sequentially writing the data to be processed extracted by each polling into the tail part of a preset secondary cache queue.
Optionally, the second-level buffer queue is an annular buffer queue;
the writing of the data to be processed extracted by each polling into a preset second-level cache queue in sequence comprises the following steps:
and sequentially writing the data to be processed extracted by each polling into the head of a preset second-level cache queue.
Optionally, the sequentially writing the data to be processed extracted by each polling into a preset second-level cache queue includes:
and sequentially writing the data to be processed extracted by each polling into a preset second-level cache queue from high to low according to the priority.
In a second aspect, an embodiment of the present invention further provides a data processing system, where the data processing system includes a data production module for producing data and a data processing module for consuming data, and the data production module includes:
a production unit for producing at least one data to be processed;
the priority setting unit is used for setting the priority for the at least one piece of data to be processed;
a first-level queue writing unit, configured to write the at least one piece of data to be processed into a first-level buffer queue in a preset first-level buffer queue set corresponding to a priority, according to the priority of the at least one piece of data to be processed; the first-level cache queue set is used for placing data to be processed with the same priority;
the data extraction unit is used for sequentially extracting a preset number of data to be processed corresponding to a corresponding primary cache queue from the primary cache queues in each primary cache queue set according to a queue polling mode until at least one data to be processed is extracted;
and the second-level queue writing unit is used for sequentially writing the data to be processed extracted by each polling into a preset second-level cache queue so that the data processing module extracts the data to be processed from the second-level cache queue for processing.
Optionally, the first-level cache queue is a single congestion queue;
the first-level queue writing unit is specifically configured to:
writing the at least one piece of data to be processed into the tail part of a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed;
the data extraction unit is specifically configured to:
and sequentially extracting a preset number of data to be processed corresponding to the corresponding first-level cache queue from the head of the first-level cache queue in each first-level cache queue set according to a queue polling mode until at least one data to be processed is extracted.
Optionally, the first-level buffer queue is an annular buffer queue;
the first-level queue writing unit is specifically configured to:
writing the at least one piece of data to be processed into the head of a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed;
the data extraction unit is specifically configured to:
and sequentially extracting a preset amount of data to be processed corresponding to the corresponding first-level cache queue from the tail part of the first-level cache queue in each first-level cache queue set according to a queue polling mode until at least one data to be processed is extracted.
Optionally, the second-level cache queue is a single congestion queue;
the secondary queue write unit is specifically configured to:
and sequentially writing the data to be processed extracted by each polling into the tail part of a preset secondary cache queue so that the data to be processed is extracted from the secondary cache queue by the data processing module for processing.
Optionally, the second-level buffer queue is an annular buffer queue;
the secondary queue write unit is specifically configured to:
and sequentially writing the data to be processed extracted by each polling into the head of a preset second-level cache queue so that the data processing module extracts the data to be processed from the second-level cache queue for processing.
Optionally, the secondary queue writing unit is specifically configured to:
and sequentially writing the to-be-processed data extracted by each polling into a preset second-level cache queue according to the sequence of the priorities from high to low so that the data processing module extracts the to-be-processed data from the second-level cache queue for processing.
In the embodiment of the invention, the data production module produces at least one piece of data to be processed; setting a priority for at least one piece of data to be processed; writing at least one piece of data to be processed into a first-level cache queue in a first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed; sequentially extracting a preset number of data to be processed corresponding to a corresponding primary cache queue from the primary cache queues in each primary cache queue set according to a queue polling mode until at least one data to be processed is extracted; and sequentially writing the data to be processed extracted by each polling into a preset second-level cache queue so that the data processing module extracts the data to be processed from the second-level cache queue for processing. Therefore, in the scheme, the data to be processed polled from the first-level cache queues corresponding to different priorities at each time is written into the second-level cache queues, so that the data to be processed with different priorities are processed, and therefore, the data with high priority can be processed as soon as possible without blocking the queue with first-in first-out to wait for the previous data to be processed.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a data processing system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Embodiments of the present invention provide a data processing method and apparatus, so as to ensure that data with a high priority can be processed as soon as possible without blocking a queue with a first-in first-out queue for waiting for the previous data to be processed.
First, a data processing method provided in an embodiment of the present invention is described below.
It should be noted that the data processing method provided by the embodiment of the present invention is applied to a data processing system, where the data processing system includes a data production module for producing data and a data processing module for processing data; in practical applications, the types of the data production module and the data processing module may be specifically a business system, a class, a function, a thread, a process, or the like.
And, two levels of queues are preset: the system comprises a first-level cache queue set and a second-level cache queue set, wherein the first-level cache queue set is used for placing data to be processed with the same priority, the second-level cache queue set is used for placing data to be processed with different priorities, and the first-level cache queue set comprises at least one first-level cache queue. In practical applications, the first-level buffer queue and the second-level buffer queue may be a single congestion queue or a circular buffer queue, but is not limited thereto.
It is understood that the number of sets of level one cache queues may be the same as the number of priorities; for the single congestion queue, data is written from the tail part, and data is extracted from the head part; for the circular buffer queue, data is written from the head part, and data is extracted from the tail part; moreover, both the single congestion queue and the circular buffer queue accord with the first-in first-out principle.
As shown in fig. 1, a data processing method may include:
s101, the data production module produces at least one piece of data to be processed;
s102, setting a priority for at least one piece of data to be processed;
after the data production module produces at least one piece of data to be processed, priority can be set for the at least one piece of data to be processed so as to distinguish processing urgency of different pieces of data to be processed.
S103, writing at least one piece of data to be processed into a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed;
after the priority is set on the data to be processed, writing the data to be processed into a first-level buffer queue in a preset first-level buffer queue set corresponding to the priority according to the priority of the data to be processed.
It should be noted that the first-level buffer queue set may include a first-level buffer queue, that is, one priority corresponds to one first-level buffer queue; alternatively, the set of level one buffer queues may include at least two level one buffer queues, that is, one priority level corresponds to at least two level one buffer queues. For the case that one priority corresponds to one first-level cache queue, the data to be processed with the same priority can be directly written into the first-level cache queue corresponding to the priority. For the case that one priority corresponds to at least two first-level buffer queues, to-be-processed data with the same priority may be sequentially written into different first-level buffer queues corresponding to the priority according to a preset writing rule, for example: the data to be processed with the same priority level is polled and written into different first-level cache queues corresponding to the priority level in a mode of writing one data to be processed each time, or the data to be processed with the same priority level is randomly divided into a plurality of groups and then the data to be processed in each group is written into the corresponding first-level cache queue corresponding to the priority level, and the like.
For example, the priority of the data to be processed includes: p1, p2 and p3 … pn, the data to be processed of the priority p1 can be written into the first level buffer queue Q1 in the corresponding first level buffer queue set, the data to be processed of the priority p2 can be written into the first level buffer queue Q2 in the corresponding first level buffer queue set, and so on, the data to be processed of the priority pn can be written into the first level buffer queue Qn in the corresponding first level buffer queue set. And if the data of a certain priority px does not exist, not writing the data into the first-level cache queue Qx in the corresponding first-level cache queue set.
S104, sequentially extracting a preset number of data to be processed corresponding to a corresponding primary cache queue from the primary cache queues in each primary cache queue set according to a queue polling mode until at least one data to be processed is extracted;
after writing at least one piece of data to be processed into a first-level buffer queue in a first-level buffer queue set, sequentially extracting a preset number of pieces of data to be processed corresponding to a corresponding first-level buffer queue from the first-level buffer queues in each first-level buffer queue set according to a queue polling mode until the at least one piece of data to be processed is extracted, so as to ensure that the priority of the data to be processed extracted by polling each time comprises the priority of all the data to be processed currently in the first-level buffer queues. It can be understood that the data to be processed in each first-level buffer queue is continuously fetched by means of queue polling, and since each polling will fetch a certain amount of data to be processed, if there is data to be processed in the first-level buffer queue with high priority, the data to be processed must be fetched in the current polling.
Specifically, for the case that the first-level buffer queue is a single congestion queue, at least one piece of data to be processed may be written into the tail of the first-level buffer queue in the preset first-level buffer queue set corresponding to the priority, according to the priority of the at least one piece of data to be processed; correspondingly, a preset amount of data to be processed corresponding to the corresponding first-level cache queue can be sequentially extracted from the head of the first-level cache queue in each first-level cache queue set according to a queue polling mode.
Specifically, for the first-level buffer queue being an annular buffer queue, at least one to-be-processed data may be written into a head of the first-level buffer queue in the first-level buffer queue set corresponding to the priority, according to the priority of the at least one to-be-processed data; correspondingly, according to a queue polling mode, sequentially extracting a preset amount of to-be-processed data corresponding to the corresponding first-level cache queue from the tail part of the first-level cache queue in each first-level cache queue set.
Moreover, it should be noted that, on the premise of ensuring that the low-priority data of the first-level buffer queue is not completely fetched at one time:
when each first-level cache queue set comprises one first-level cache queue, the preset number can be one, two or three;
or,
when each first-level cache queue set comprises at least one first-level cache queue, the preset number can be one, two or three;
or,
when each first-level cache queue set includes one first-level cache queue, the preset number may be a number determined according to an empirical value method or a dynamic calculation method.
For the case that each set of level one buffer queues includes at least one level one buffer queue, for example: the first-level buffer queue Q1 in the first-level buffer queue set corresponding to the priority p1 contains n1 queues, the first-level buffer queue Q2 in the first-level buffer queue set corresponding to the priority p2 contains n2 queues, and the first-level buffer queue Q2 in the first-level buffer queue set corresponding to the priority p3 contains n3 queues, at this time, n1 pieces of data of the priority p1, n2 pieces of data of the priority p2, and n3 pieces of data of the priority p3 can be extracted each time, so that the data of each priority can be processed without being blocked.
Moreover, it should be noted that the preset number is estimated by an empirical value method according to the priority of the data to be processed and the corresponding data size; the dynamic calculation method is to dynamically calculate according to the response speed required by data of a certain priority and the actual data quantity, so as to ensure that the data of high priority is inevitably processed in the data processing module processing a certain quantity of data.
And S105, sequentially writing the data to be processed extracted in each polling into a preset second-level cache queue so that the data processing module extracts the data to be processed from the second-level cache queue for processing.
After each polling is finished, or after all the data to be processed are extracted, in order to ensure that the data with high priority can be processed as soon as possible without blocking the queue of first-in first-out and waiting for the previous data to be processed, the data to be processed extracted by each polling is sequentially written into a preset second-level cache queue, so that the data processing module extracts the data to be processed from the second-level cache queue for processing. Specifically, the data to be processed extracted in each polling may be written into a preset second-level cache queue in the order from high to low in priority; of course, the data to be processed extracted by each polling may be randomly arranged and then sequentially written into the preset second-level buffer queue, which is reasonable.
Specifically, for the case that the second-level buffer queue may be a single-blocking queue, the data to be processed extracted by polling each time may be written into the tail of the preset second-level buffer queue in sequence; accordingly, the data processing module can extract the data to be processed in the second-level buffer queue from the head of the second-level buffer queue.
Specifically, for the case that the second-level buffer queue is an annular buffer queue, the data to be processed extracted by each polling may be sequentially written into the head of the preset second-level buffer queue; accordingly, the data processing module can extract the data to be processed in the second-level buffer queue from the tail part of the second-level buffer queue.
In the embodiment of the invention, the data production module produces at least one piece of data to be processed; setting a priority for at least one piece of data to be processed; writing at least one piece of data to be processed into a first-level cache queue in a first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed; sequentially extracting a preset number of data to be processed corresponding to a corresponding primary cache queue from the primary cache queues in each primary cache queue set according to a queue polling mode until at least one data to be processed is extracted; and sequentially writing the data to be processed extracted by each polling into a preset second-level cache queue so that the data processing module extracts the data to be processed from the second-level cache queue for processing. Therefore, in the scheme, the data to be processed polled from the first-level cache queues corresponding to different priorities at each time is written into the second-level cache queues, so that the data to be processed with different priorities are processed, and therefore, the data with high priority can be processed as soon as possible without blocking the queue with first-in first-out to wait for the previous data to be processed.
The data processing method provided by the embodiment of the invention is described below with reference to specific application examples.
Suppose there are two levels of priority for the data to be processed: p1 and P2, wherein P2 is higher than P1; a first-level buffer queue Q1 and a first-level buffer queue Q2 belonging to a single congestion queue are created in advance, wherein Q1 is used for placing data to be processed with a priority P1, and Q2 is used for placing data to be processed with a priority P2; and, a second level buffer queue belonging to the single congestion queue is created in advance.
(1) The data production module produces low priority P1 pending data for a certain period of time: a1, a2, A3, a4, a5, a6, a7, A8, high priority P2: b1, B2, B3, B4.
(2) The data production module first writes a1, a2, A3, a4, a5, a6, a7, A8 to the tail of Q1 in sequence, and writes B1, B2, B3, B4 to the tail of Q2 in sequence:
wherein, Q1 is [ A1, A2, A3, A4, A5, A6, A7, A8 ];
Q2:[B1,B2,B3,B4]。
(3) through a queue polling mode, the data production module sequentially extracts a piece of data to be processed from the heads of the Q1 and the Q2, rearranges the extracted data to be processed and writes the rearranged data to the tail of the secondary buffer queue. Specifically, the method comprises the following steps:
poll 1, extract [ a1, B1], level two buffer queue: [ A1, B1 ];
poll 2, extract [ a2, B2], level two buffer queue: [ A1, B1, A2, B2 ];
poll 3, extract [ a3, B3], level two buffer queue: [ A1, B1, A2, B2, A3, B3 ];
poll 4, extract [ a4, B4], level two buffer queue: [ A1, B1, A2, B2, A3, B3, A4, B4 ];
poll 5, extract [ a5], level two buffer queue: [ A1, B1, A2, B2, A3, B3, A4, B4, A5 ];
poll 6, extract [ a6], level two buffer queue: [ A1, B1, A2, B2, A3, B3, A4, B4, A5, A6 ];
poll 7, extract [ a7], level two buffer queue: [ A1, B1, A2, B2, A3, B3, A4, B4, A5, A6, A7 ];
polling 8, extract [ A8], level two buffer queue: [ A1, B1, A2, B2, A3, B3, A4, B4, A5, A6, A7, A8 ].
Further, the data processing module may fetch the data to be processed at the head of the second-level buffer queue, and sequentially process the data according to the order of a1, B1, a2, B2, A3, B3, a4, B4, a5, A6, a7, and A8.
Therefore, the scheme can ensure that the data with high priority can be processed as soon as possible without blocking the queue waiting for the previous data to be processed in the first-in first-out queue.
Corresponding to the above method embodiment, as shown in fig. 2, an embodiment of the present invention provides a data processing system, where the data processing system includes a data production module 210 for producing data and a data processing module 220 for processing data, and the data production module may include:
a production unit 211 for producing at least one data to be processed;
a priority setting unit 212, configured to set a priority for the at least one piece of data to be processed;
a first-level queue writing unit 213, configured to write the at least one piece of data to be processed into a first-level buffer queue in a preset first-level buffer queue set corresponding to a priority level of the at least one piece of data to be processed according to the priority level of the at least one piece of data to be processed; the first-level cache queue set is used for placing data to be processed with the same priority;
a data extracting unit 214, configured to sequentially extract, according to a queue polling manner, a preset number of to-be-processed data corresponding to a corresponding primary cache queue from the primary cache queues in each primary cache queue set until at least one of the to-be-processed data is extracted;
the second-level queue writing unit 215 is configured to sequentially write the to-be-processed data extracted by each polling into a preset second-level buffer queue, so that the data processing module 220 extracts the to-be-processed data from the second-level buffer queue for processing.
In the embodiment of the invention, the data production module produces at least one piece of data to be processed; setting a priority for at least one piece of data to be processed; writing at least one piece of data to be processed into a first-level cache queue in a first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed; sequentially extracting a preset number of data to be processed corresponding to a corresponding primary cache queue from the primary cache queues in each primary cache queue set according to a queue polling mode until at least one data to be processed is extracted; and sequentially writing the data to be processed extracted by each polling into a preset second-level cache queue so that the data processing module extracts the data to be processed from the second-level cache queue for processing. Therefore, in the scheme, the data to be processed polled from the first-level cache queues corresponding to different priorities at each time is written into the second-level cache queues, so that the data to be processed with different priorities are processed, and therefore, the data with high priority can be processed as soon as possible without blocking the queue with first-in first-out to wait for the previous data to be processed.
Specifically, the first-level cache queue is a single blocking queue;
the first-level queue writing unit 213 is specifically configured to:
writing the at least one piece of data to be processed into the tail part of a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed;
the data extracting unit 214 is specifically configured to:
and sequentially extracting a preset number of data to be processed corresponding to the corresponding first-level cache queue from the head of the first-level cache queue in each first-level cache queue set according to a queue polling mode until at least one data to be processed is extracted.
Specifically, the first-level cache queue is an annular cache queue;
the first-level queue writing unit 213 is specifically configured to:
writing the at least one piece of data to be processed into the head of a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed;
the data extracting unit 214 is specifically configured to:
and sequentially extracting a preset amount of data to be processed corresponding to the corresponding first-level cache queue from the tail part of the first-level cache queue in each first-level cache queue set according to a queue polling mode until at least one data to be processed is extracted.
Specifically, the second-level cache queue is a single congestion queue;
the secondary queue writing unit 215 is specifically configured to:
and sequentially writing the data to be processed extracted by each polling into the tail part of a preset secondary cache queue so that the data to be processed is extracted from the secondary cache queue by the data processing module for processing.
Specifically, the second-level cache queue is a ring-shaped cache queue;
the secondary queue writing unit 215 is specifically configured to:
and sequentially writing the data to be processed extracted by each polling into the head of a preset second-level cache queue so that the data processing module extracts the data to be processed from the second-level cache queue for processing.
Specifically, the secondary queue writing unit 215 is specifically configured to:
and sequentially writing the to-be-processed data extracted by each polling into a preset second-level cache queue according to the sequence of the priorities from high to low so that the data processing module extracts the to-be-processed data from the second-level cache queue for processing.
For system or apparatus embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for relevant points.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
Those skilled in the art will appreciate that all or part of the steps in the above method embodiments may be implemented by a program to instruct relevant hardware to perform the steps, and the program may be stored in a computer-readable storage medium, which is referred to herein as a storage medium, such as: ROM/RAM, magnetic disk, optical disk, etc.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (12)
1. The data processing method is applied to a data processing system, wherein the data processing system comprises a data production module for producing data and a data processing module for processing the data; the method comprises the following steps:
the data production module produces at least one piece of data to be processed;
setting a priority for the at least one piece of data to be processed;
writing the at least one piece of data to be processed into a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed; the first-level cache queue set is used for placing data to be processed with the same priority;
sequentially extracting a preset number of data to be processed corresponding to a corresponding primary cache queue from the primary cache queues in each primary cache queue set according to a queue polling mode until at least one data to be processed is extracted;
and sequentially writing the data to be processed extracted by each polling into a preset second-level cache queue so that the data to be processed is extracted from the second-level cache queue by the data processing module for processing.
2. The method of claim 1, wherein the level one cache queue is a single congestion queue;
the writing of the at least one piece of data to be processed into a first-level buffer queue in a preset first-level buffer queue set corresponding to the priority according to the priority of the at least one piece of data to be processed includes:
writing the at least one piece of data to be processed into the tail part of a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed;
according to the queue polling mode, sequentially extracting the data to be processed with the preset quantity corresponding to the corresponding first-level cache queue from the first-level cache queues in each first-level cache queue set, and the method comprises the following steps:
and sequentially extracting a preset amount of data to be processed corresponding to the corresponding first-level cache queue from the head of the first-level cache queue in each first-level cache queue set according to a queue polling mode.
3. The method of claim 1, wherein the level one buffer queue is a circular buffer queue;
the writing of the at least one piece of data to be processed into a first-level buffer queue in a preset first-level buffer queue set corresponding to the priority according to the priority of the at least one piece of data to be processed includes:
writing the at least one piece of data to be processed into the head of a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed;
according to the queue polling mode, sequentially extracting the data to be processed with the preset quantity corresponding to the corresponding first-level cache queue from the first-level cache queues in each first-level cache queue set, and the method comprises the following steps:
and sequentially extracting a preset amount of data to be processed corresponding to the corresponding first-level cache queue from the tail part of the first-level cache queue in each first-level cache queue set according to a queue polling mode.
4. The method of any of claims 1-3, wherein the secondary cache queue is a single congestion queue;
the writing of the data to be processed extracted by each polling into a preset second-level cache queue in sequence comprises the following steps:
and sequentially writing the data to be processed extracted by each polling into the tail part of a preset secondary cache queue.
5. The method according to any one of claims 1-3, wherein the secondary buffer queue is a circular buffer queue;
the writing of the data to be processed extracted by each polling into a preset second-level cache queue in sequence comprises the following steps:
and sequentially writing the data to be processed extracted by each polling into the head of a preset second-level cache queue.
6. The method according to any one of claims 1 to 3, wherein sequentially writing the data to be processed extracted by each polling to a preset second level buffer queue comprises:
and sequentially writing the data to be processed extracted by each polling into a preset second-level cache queue from high to low according to the priority.
7. A data processing system comprising a data production module for producing data and a data processing module for consuming data, the data production module comprising:
a production unit for producing at least one data to be processed;
the priority setting unit is used for setting the priority for the at least one piece of data to be processed;
a first-level queue writing unit, configured to write the at least one piece of data to be processed into a first-level buffer queue in a preset first-level buffer queue set corresponding to a priority, according to the priority of the at least one piece of data to be processed; the first-level cache queue set is used for placing data to be processed with the same priority;
the data extraction unit is used for sequentially extracting a preset number of data to be processed corresponding to a corresponding primary cache queue from the primary cache queues in each primary cache queue set according to a queue polling mode until at least one data to be processed is extracted;
and the second-level queue writing unit is used for sequentially writing the data to be processed extracted by each polling into a preset second-level cache queue so that the data processing module extracts the data to be processed from the second-level cache queue for processing.
8. The system of claim 7, wherein the level one cache queue is a single congestion queue;
the first-level queue writing unit is specifically configured to:
writing the at least one piece of data to be processed into the tail part of a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed;
the data extraction unit is specifically configured to:
and sequentially extracting a preset number of data to be processed corresponding to the corresponding first-level cache queue from the head of the first-level cache queue in each first-level cache queue set according to a queue polling mode until at least one data to be processed is extracted.
9. The system of claim 7, wherein the level one buffer queue is a circular buffer queue;
the first-level queue writing unit is specifically configured to:
writing the at least one piece of data to be processed into the head of a first-level cache queue in a preset first-level cache queue set corresponding to the priority according to the priority of the at least one piece of data to be processed;
the data extraction unit is specifically configured to:
and sequentially extracting a preset amount of data to be processed corresponding to the corresponding first-level cache queue from the tail part of the first-level cache queue in each first-level cache queue set according to a queue polling mode until at least one data to be processed is extracted.
10. The system of any of claims 7-9, wherein the secondary cache queue is a single congestion queue;
the secondary queue write unit is specifically configured to:
and sequentially writing the data to be processed extracted by each polling into the tail part of a preset secondary cache queue so that the data to be processed is extracted from the secondary cache queue by the data processing module for processing.
11. The system according to any one of claims 7-9, wherein the secondary buffer queue is a circular buffer queue;
the secondary queue write unit is specifically configured to:
and sequentially writing the data to be processed extracted by each polling into the head of a preset second-level cache queue so that the data processing module extracts the data to be processed from the second-level cache queue for processing.
12. The system according to any of claims 7-9, wherein the secondary queue write unit is specifically configured to:
and sequentially writing the to-be-processed data extracted by each polling into a preset second-level cache queue according to the sequence of the priorities from high to low so that the data processing module extracts the to-be-processed data from the second-level cache queue for processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410415527.0A CN104199790B (en) | 2014-08-21 | 2014-08-21 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410415527.0A CN104199790B (en) | 2014-08-21 | 2014-08-21 | Data processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104199790A true CN104199790A (en) | 2014-12-10 |
| CN104199790B CN104199790B (en) | 2017-10-24 |
Family
ID=52085086
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410415527.0A Active CN104199790B (en) | 2014-08-21 | 2014-08-21 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104199790B (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105760236A (en) * | 2016-05-13 | 2016-07-13 | 中国工商银行股份有限公司 | Data collection method and system of distributed computer cluster |
| CN106569746A (en) * | 2016-11-01 | 2017-04-19 | 北京信安世纪科技有限公司 | Data writing method and device |
| CN107171918A (en) * | 2017-04-26 | 2017-09-15 | 成都成电光信科技股份有限公司 | Support the messaging method in the GJB289A bus modules of priority |
| CN109471731A (en) * | 2018-11-21 | 2019-03-15 | 阿里巴巴集团控股有限公司 | A kind of data processing, EMS memory management process, device, equipment and medium |
| CN110019358A (en) * | 2017-09-30 | 2019-07-16 | 北京搜狗科技发展有限公司 | A kind of data processing method, device and equipment and storage medium |
| CN110673786A (en) * | 2019-09-03 | 2020-01-10 | 浪潮电子信息产业股份有限公司 | Data caching method and device |
| CN111355673A (en) * | 2018-12-24 | 2020-06-30 | 深圳市中兴微电子技术有限公司 | Data processing method, device, equipment and storage medium |
| CN111865741A (en) * | 2019-04-24 | 2020-10-30 | 瑞昱半导体股份有限公司 | Data transmission method and data transmission system |
| CN114500403A (en) * | 2022-01-24 | 2022-05-13 | 中国联合网络通信集团有限公司 | Data processing method and device and computer readable storage medium |
| CN114721823A (en) * | 2022-03-29 | 2022-07-08 | 中信百信银行股份有限公司 | Information processing method, device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6728265B1 (en) * | 1999-07-30 | 2004-04-27 | Intel Corporation | Controlling frame transmission |
| CN1508709A (en) * | 2002-11-25 | 2004-06-30 | ��ʽ���������Ƽ� | Judging circuit and data processing system |
| CN1581835A (en) * | 2003-07-31 | 2005-02-16 | 富士通株式会社 | Media access control device for wireless LAN |
| CN101478472A (en) * | 2008-10-21 | 2009-07-08 | 北京闪联讯通数码科技有限公司 | Socket data transmission processing method and apparatus |
| CN103488574A (en) * | 2012-06-12 | 2014-01-01 | Ls产电株式会社 | Circuit for memory sharing |
| US20140185451A1 (en) * | 2012-12-31 | 2014-07-03 | Telefonaktiebolaget L M Ericsson (Publ) | Low pass filter for hierarchical pipelined distributed scheduling traffic manager |
-
2014
- 2014-08-21 CN CN201410415527.0A patent/CN104199790B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6728265B1 (en) * | 1999-07-30 | 2004-04-27 | Intel Corporation | Controlling frame transmission |
| CN1508709A (en) * | 2002-11-25 | 2004-06-30 | ��ʽ���������Ƽ� | Judging circuit and data processing system |
| CN1581835A (en) * | 2003-07-31 | 2005-02-16 | 富士通株式会社 | Media access control device for wireless LAN |
| CN101478472A (en) * | 2008-10-21 | 2009-07-08 | 北京闪联讯通数码科技有限公司 | Socket data transmission processing method and apparatus |
| CN103488574A (en) * | 2012-06-12 | 2014-01-01 | Ls产电株式会社 | Circuit for memory sharing |
| US20140185451A1 (en) * | 2012-12-31 | 2014-07-03 | Telefonaktiebolaget L M Ericsson (Publ) | Low pass filter for hierarchical pipelined distributed scheduling traffic manager |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105760236A (en) * | 2016-05-13 | 2016-07-13 | 中国工商银行股份有限公司 | Data collection method and system of distributed computer cluster |
| CN106569746A (en) * | 2016-11-01 | 2017-04-19 | 北京信安世纪科技有限公司 | Data writing method and device |
| CN107171918B (en) * | 2017-04-26 | 2020-06-16 | 成都成电光信科技股份有限公司 | Message transceiving method in GJB289A bus module supporting priority |
| CN107171918A (en) * | 2017-04-26 | 2017-09-15 | 成都成电光信科技股份有限公司 | Support the messaging method in the GJB289A bus modules of priority |
| CN110019358A (en) * | 2017-09-30 | 2019-07-16 | 北京搜狗科技发展有限公司 | A kind of data processing method, device and equipment and storage medium |
| CN110019358B (en) * | 2017-09-30 | 2021-08-24 | 北京搜狗科技发展有限公司 | Data processing method, device and equipment and storage medium |
| CN109471731A (en) * | 2018-11-21 | 2019-03-15 | 阿里巴巴集团控股有限公司 | A kind of data processing, EMS memory management process, device, equipment and medium |
| CN111355673A (en) * | 2018-12-24 | 2020-06-30 | 深圳市中兴微电子技术有限公司 | Data processing method, device, equipment and storage medium |
| WO2020134425A1 (en) * | 2018-12-24 | 2020-07-02 | 深圳市中兴微电子技术有限公司 | Data processing method, apparatus, and device, and storage medium |
| CN111865741A (en) * | 2019-04-24 | 2020-10-30 | 瑞昱半导体股份有限公司 | Data transmission method and data transmission system |
| CN110673786A (en) * | 2019-09-03 | 2020-01-10 | 浪潮电子信息产业股份有限公司 | Data caching method and device |
| US11803475B2 (en) | 2019-09-03 | 2023-10-31 | Inspur Electronic Information Industry Co., Ltd. | Method and apparatus for data caching |
| CN114500403A (en) * | 2022-01-24 | 2022-05-13 | 中国联合网络通信集团有限公司 | Data processing method and device and computer readable storage medium |
| CN114721823A (en) * | 2022-03-29 | 2022-07-08 | 中信百信银行股份有限公司 | Information processing method, device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104199790B (en) | 2017-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104199790B (en) | Data processing method and device | |
| US20190155656A1 (en) | Method and system for scheduling threads for execution | |
| WO2019083673A3 (en) | Command selection policy | |
| US10325219B2 (en) | Parallel retrieval of training data from multiple producers for machine learning systems | |
| CN113282659A (en) | Data processing method and device based on block chain | |
| EP4113299A2 (en) | Task processing method and device, and electronic device | |
| US10922134B2 (en) | Method, device and computer program product for processing data | |
| CN106909554B (en) | Method and device for loading database text table data | |
| CN105262680A (en) | Multi-threaded NAS Gateway applied to cloud storage system | |
| CN112579263A (en) | Task execution method and device, storage medium and electronic equipment | |
| US10310891B2 (en) | Hand-off scheduling | |
| US8621479B2 (en) | System and method for selecting task allocation method based on load balancing and core affinity metrics | |
| CN111124708B (en) | Microservice-oriented batch reasoning method, server and computer readable storage medium | |
| US9886320B2 (en) | Method for prioritizing tasks queued at a server system | |
| CN107239343B (en) | Data processing method and device | |
| CN107391730B (en) | SQL statement processing method and device | |
| CN112860401A (en) | Task scheduling method and device, electronic equipment and storage medium | |
| EP2905703A1 (en) | Parallel computer system, control method of parallel computer system, and computer-readable storage medium | |
| JP2002140208A (en) | Performance simulator, performance simulation method, and recording medium with performance simulation program recorded thereon | |
| US20200301732A1 (en) | Information processing system and non-transitory computer readable medium storing program | |
| US9766940B2 (en) | Enabling dynamic job configuration in mapreduce | |
| WO2016082463A1 (en) | Data processing method and apparatus for multi-core processor, and storage medium | |
| CN102638403A (en) | Method and device for processing messages | |
| EP3679468A1 (en) | Controller event queues | |
| CN114358577A (en) | Order information processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |