CN112463416A - Method, device, equipment and storage medium for processing data in message queue - Google Patents
Method, device, equipment and storage medium for processing data in message queue Download PDFInfo
- Publication number
- CN112463416A CN112463416A CN202011494838.2A CN202011494838A CN112463416A CN 112463416 A CN112463416 A CN 112463416A CN 202011494838 A CN202011494838 A CN 202011494838A CN 112463416 A CN112463416 A CN 112463416A
- Authority
- CN
- China
- Prior art keywords
- data
- message queue
- time
- alarm
- consumer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000012545 processing Methods 0.000 title claims abstract description 37
- 238000013500 data storage Methods 0.000 claims abstract description 44
- 230000014759 maintenance of location Effects 0.000 claims description 26
- 238000004590 computer program Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 description 45
- 230000007246 mechanism Effects 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000004140 cleaning Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000005192 partition Methods 0.000 description 3
- 238000003672 processing method Methods 0.000 description 3
- 230000003111 delayed effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a method, a device, equipment and a storage medium for processing data in a message queue. The method comprises the following steps: acquiring offset information of data in a message queue of a target topic and consumption progress of a data consumer for the data in the message queue of the target topic, wherein the offset information of the data comprises an offset of the data in the message queue in at least one statistical period, and the consumption progress comprises the offset of the data consumed by the data consumer in the message queue; determining a lag time of the data consumer according to the offset information and the consumption progress; and adjusting the data storage time of the message queue according to the lag time. The method reduces the risk of data loss.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a storage medium for processing data in a message queue.
Background
With the development of information technology, there is a need for data sharing between application systems within an enterprise. kafka is a widely used distributed message queue system at present, and the requirement of data sharing can be realized through kafka. In particular, a message subscriber (data consumer) to kafka will typically consume data produced by a message producer (data provider) from a partition in a class of topics.
In some scenarios, the data consumer may consume the data at a slower rate than the data provider produces the data, which in turn may lead to backlogs of data in the partition. When the data in the partition exceeds the preset preservable time, the cleaning mechanism of kafka is triggered, thereby causing data loss.
Thus, there is a risk of data loss when the data consumer consumes data at a slower rate than the data provider produces data.
Disclosure of Invention
In order to solve the technical problem, the application provides a method, a device, equipment and a storage medium for processing data in a message queue, and the method can avoid the problem of data loss caused by a cleaning mechanism of kafka.
The embodiment of the application discloses the following technical scheme:
in a first aspect, the present application provides a method for processing data in a message queue, including:
acquiring offset information of data in a message queue of a target topic and consumption progress of a data consumer for the data in the message queue of the target topic, wherein the offset information of the data comprises an offset of the data in the message queue in at least one statistical period, and the consumption progress comprises the offset of the data consumed by the data consumer in the message queue;
determining a lag time of the data consumer according to the offset information and the consumption progress;
and adjusting the data storage time of the message queue according to the lag time.
Optionally, the method further includes: and when the lag time is larger than or equal to an alarm time threshold value, feeding back alarm information to the data consumer.
Optionally, the method further includes:
determining a lag data amount of the data consumer according to the offset information and the consumption progress;
and when the hysteresis data volume is larger than or equal to the alarm data volume threshold value, feeding back alarm information to the data consumer.
Optionally, the method further includes:
acquiring alarm configuration information of a data consumer;
and generating an alarm condition according to the alarm configuration information, wherein the alarm condition is used for triggering an alarm process when being met.
Optionally, the adjusting the data saving time of the message queue according to the lag time includes:
when the lag time is greater than or equal to a lag time threshold, adjusting the data retention time of the message queue;
or the like, or, alternatively,
adjusting the data retention time of the message queue when the difference between the lag time and the data retention time of the message queue is less than or equal to a remaining time threshold.
Optionally, the adjusting the data saving time of the message queue includes:
if the sum of the lag time and the data storage time of the message queue is greater than or equal to the maximum data storage time of the message queue, adjusting the data storage time to the maximum data storage time;
and if the sum of the lag time and the data storage time of the message queue is less than the maximum data storage time of the message queue, adjusting the data storage time to the sum of the data storage time and the lag time.
In a second aspect, the present application provides an apparatus for processing data in a message queue, including: the device comprises an acquisition module, a determination module and an adjustment module;
the obtaining module is configured to obtain offset information of data in a message queue of a target topic and a consumption progress of a data consumer for the data in the message queue of the target topic, where the offset information of the data includes an offset of the data in the message queue in at least one statistical period, and the consumption progress includes an offset of the data consumed by the data consumer in the message queue;
the determining module is used for determining the lag time of the data consumer according to the offset information and the consumption progress;
and the adjusting module is used for adjusting the data storage time of the message queue according to the lag time.
Optionally, the apparatus further comprises: an alarm module;
and the alarm module is used for feeding back alarm information to the data consumer when the lag time is greater than or equal to an alarm time threshold.
Optionally, the apparatus further comprises: an alarm module;
the determining module is further used for determining the lag data volume of the data consumer according to the offset information and the consumption progress;
and the alarm module is used for feeding back alarm information to the data consumer when the hysteresis data volume is greater than or equal to an alarm data volume threshold value.
Optionally, the obtaining module is further configured to obtain alarm configuration information of the data consumer;
and generating an alarm condition according to the alarm configuration information, wherein the alarm condition is used for triggering an alarm process when being met.
Optionally, the adjusting module is specifically configured to adjust the data storage time of the message queue when the lag time is greater than or equal to a lag time threshold;
or the like, or, alternatively,
adjusting the data retention time of the message queue when the difference between the lag time and the data retention time of the message queue is less than or equal to a remaining time threshold.
Optionally, the adjusting module is specifically configured to adjust the data saving time to the maximum data saving time if a sum of the lag time and the data saving time of the message queue is greater than or equal to the maximum data saving time of the message queue;
and if the sum of the lag time and the data storage time of the message queue is less than the maximum data storage time of the message queue, adjusting the data storage time to the sum of the data storage time and the lag time.
In a third aspect, the present application provides a device for processing data in a message queue, where the device includes: a memory and a processor;
the memory is used for storing a computer program and transmitting the computer program to the processor;
the processor, according to instructions in the computer program, executes the method for processing data in a message queue according to any one of the first aspect.
In a third aspect, the present application provides a computer readable storage medium for storing computer software instructions for a data processing system in a message queue, which when run on a computer, enable the computer to perform the method of processing data in a message queue according to any one of the first aspect
According to the technical scheme, the method has the following beneficial effects:
the method obtains offset information of data in a message queue of a target theme and consumption progress of a data consumer on the data in the message queue of the target theme. The offset information comprises the offset of the data in the message queue in at least one statistical period, the message progress comprises the offset of the data consumed by the data consumer in the message queue, and the method can determine the lag time of the data consumer according to the offset information and the consumption progress, namely the lag time is the lag time of the data in the message queue consumed by the data consumer and the latest data in the message queue, and then adjust the data storage time of the message queue according to the lag time, so that the situation that the data in the message queue is cleared due to the fact that the data storage time is exceeded is avoided. Therefore, the method can reduce the risk of data loss.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a system architecture diagram of a monitoring system according to an embodiment of the present disclosure;
fig. 2 is a schematic diagram of offset information provided in an embodiment of the present application;
fig. 3 is a flowchart of a method for processing data in a message queue according to an embodiment of the present application;
fig. 4 is a flowchart of a method for processing data in a message queue according to an embodiment of the present application;
fig. 5 is a schematic diagram of an apparatus for processing data in a message queue according to an embodiment of the present application;
fig. 6 is a schematic diagram of an apparatus for processing data in a message queue according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
For the sake of understanding, the technical terms related to the present application will be described below.
A message queue refers to a container that holds messages during the transmission of the messages. In the distributed message queue system kafka, a data provider pushes data to a topic of kafka so that a data consumer can consume the data in the topic. Generally, a data consumer will consume data according to the sequence in which the data provider writes data in the theme.
In some scenarios, there may be situations where the data consumer consumes data at a slower rate than the data provider writes the data, which in turn may lead to backlogs of data in the topic. In kafka, in order to ensure that data written by a data provider can be received, when the data in a theme exceeds a preset holdable time, the kafka triggers a cleaning mechanism. This results in some data being purged without being consumed by the data consumer, resulting in data loss and further business impact.
In view of the above, the present application provides a method for processing data in a message queue. The data processing method can be realized by a monitoring system. Specifically, the monitoring system obtains offset information of data in a message queue of a target topic and consumption progress of a data consumer for the data in the message queue of the target topic, the offset information of the data comprises an offset of the data in the message queue in at least one statistical period, the message progress comprises the offset of the data consumed by the data consumer in the message queue, then lag time of the data consumer is determined according to the offset information and the consumption progress, and then data storage time in the message queue is adjusted according to the lag time.
The method obtains offset information of data in a message queue of a target theme and consumption progress of a data consumer on the data in the message queue of the target theme. The offset information comprises the offset of the data in the message queue in at least one statistical period, the message progress comprises the offset of the data consumed by the data consumer in the message queue, and the method can determine the lag time of the data consumer according to the offset information and the consumption progress, namely the lag time is the lag time of the data in the message queue consumed by the data consumer and the latest data in the message queue, and then adjust the data storage time of the message queue according to the lag time, so that the situation that the data in the message queue is cleared due to the fact that the data storage time is exceeded is avoided. Therefore, the method can reduce the risk of data loss.
The monitoring system provided by the embodiment of the application can be used for data sharing, for example: data sharing between application systems within an enterprise. In some scenarios, the application system a may act as a data provider to push data to the topic of kafka, and the other application systems (e.g., the application system B subscribed to the topic) may act as data consumers to obtain data in the topic, thereby implementing data sharing between the application system a and the other application systems.
The monitoring system may be a software system. In particular, the monitoring system may be deployed in a computing device in the form of computer software to implement the function of adjusting the data retention time of a message queue. In some embodiments, the monitoring system may also be a hardware system. The hardware system includes a physical device having a function of adjusting a data retention time in a message queue.
The monitoring system is realized by subsystems with different functions and units with different functions. The embodiment of the present application does not limit the partitioning manner of subsystems and units in the monitoring system, and is described below with reference to an exemplary partitioning manner shown in fig. 1.
As shown in FIG. 1, the monitoring system 100 includes a communication unit 110, a hysteresis unit 120, an adjustment unit 130, an alarm unit 140, and a condition unit 150. Where the alarm element 140 and the condition element 150 are optional, the alarm element 140 and the condition element 150 may not be needed in some embodiments.
The communication unit 110 is configured to obtain offset information of data in a message queue of a target topic, where the offset information includes an offset of the data in the message queue in at least one statistical period, see fig. 2, which shows a schematic diagram of offset information, and as can be seen from fig. 2, the offset information includes offsets of a plurality of statistical periods, for example: the offset amount for 1:25 is 1000, the offset amount for 1:30 is 2000, and the offset amount for 4:00 is 12600. The communication unit 110 is further configured to obtain a consumption progress of the data consumer with respect to the data in the message queue of the target topic, where the consumption progress includes an offset of the data that has been consumed by the data consumer in the message queue. For example, the consumption progress may be an offset of the data currently consumed by the data consumer.
The lag unit 120 is configured to determine a lag time of the data consumer according to the offset information and the consumption schedule acquired by the communication unit 110, where the lag time is a time difference between a statistical time of the data being consumed by the data consumer and a current time. For example, using the offset information shown in fig. 2 as an example, if the consumption progress indicates that the offset of the data consumed by the data consumer is 2000, the statistical time of the data may be determined to be 1:30, and if the current time is 4:00, the lag time may be determined to be 150 minutes.
The adjusting unit 130 is configured to adjust the data holding time of the message queue according to the lag time. Due to the cleaning mechanism of kafka itself, when the lag time of the data in the message queue exceeds the data holding time, the data in the message queue is cleaned. In order to avoid data being cleared, the data saving time is adjusted, for example, the data saving time is delayed, before the lag time exceeds the data saving time, and then the situation that the data in the message queue is cleared can be avoided. The monitoring system 100 is thus able to reduce the risk of data loss.
In some implementations, the adjusting unit 130 can determine the relationship between the lag time and a lag time threshold, wherein the lag time threshold can be preset, and the lag time threshold is smaller than the data retention time. When the lag time is greater than or equal to the lag time threshold, adjusting the data retention time of the message queue.
In other implementations, the adjusting unit 130 can determine the relation between the lag time and a remaining time threshold, wherein the remaining time threshold can be preset, and the remaining time threshold is greater than 0. The data retention time of the message queue is adjusted when the difference between the lag time and the data retention time of the message queue is less than or equal to the remaining time threshold.
The adjusting unit 130 adjusts the data storage time according to the maximum data storage time in the message queue, where the maximum data storage time is the longest data storage time in the message queue. In some scenarios, the sum of the lag time and the data holding time of the message queue is greater than or equal to the maximum data holding time of the message queue, and the adjusting unit 130 adjusts the data holding time to the maximum data holding time. In other scenarios, the sum of the lag time and the data holding time of the message queue is less than the maximum data holding time of the message queue, and the adjusting unit 130 adjusts the data holding time to the sum of the data holding time and the lag time. Therefore, the adjusting unit 130 can prolong the data storage time of the message queue, and further avoid the clearing mechanism of kafka from clearing the data in the message queue, which is not consumed by the data consumer.
The monitoring system 100 is described below in conjunction with an alarm unit 140 and a condition unit 150.
The alarm unit 140 may be capable of determining a relationship between the lag time and an alarm time threshold, where the alarm time threshold may be predetermined, for example, the alarm time threshold may be less than or equal to the lag time threshold. Specifically, the alarm unit 140 feeds back alarm information to the data consumer when the lag time is greater than or equal to an alarm time threshold. The alert message may be too long of a lag time to prompt the data consumer. In order to enable the data consumer to obtain the consumption progress of the data consumer, the alarm information further includes the consumption progress of the data consumer, so that the data consumer can perform emergency processing after obtaining the consumption progress of the data consumer, for example, the consumption of data is accelerated.
In some implementations, the lag unit 120 is further configured to determine a lag data amount of the data consumer based on the offset information and the consumption progress. For example, taking the offset information shown in fig. 2 as an example, if the consumption progress indicates that the data consumed by the data consumer is offset by 2000, and if the current time is 4:00, the offset of the data corresponding to the current time is 12600, the hysteresis unit 120 can determine the hysteresis data amount by subtracting, that is, the hysteresis data amount obtained by 12600-2000 in this embodiment is 12400. The alarm unit 140 is capable of determining a relationship between the amount of lag data and an alarm data amount threshold, which may be preset. The alarm unit 140 feeds back alarm information to the data consumer when the amount of lag data is greater than or equal to the alarm data amount threshold. In order to enable the data consumer to obtain the consumption progress of the data consumer, the alarm information further includes the consumption progress of the data consumer, so that the data consumer can perform emergency processing after obtaining the consumption progress of the data consumer, for example, the consumption of data is accelerated.
In some scenarios, different data consumers may be more or less sensitive to different alert types. For example, data consumer A is more sensitive to the lag time of data in the message queue, and data consumer B is more sensitive to the lag data volume of data in the message queue. Different alarm conditions need to be determined from alarm configuration information of different data consumers. The alarm configuration information is used to describe how sensitive the data consumer is to different alarm types. Thus, the communication unit 110 is also used to obtain alarm configuration information for the data consumer. The condition unit 150 generates an alarm condition for triggering the alarm unit 140 to perform an alarm when the alarm condition is satisfied, based on the alarm configuration information. For example, when the alarm condition is satisfied, the alarm unit 140 feeds back alarm information to the data consumer.
Next, in order to make the technical solution of the present application clearer and easier to understand, the following describes in detail a method for processing data in a message queue provided in the embodiment of the present application from the perspective of the monitoring system 100.
Referring to fig. 3, a flow chart of a method of data processing in a message queue is shown, the method comprising:
s302: the monitoring system 100 obtains offset information of the data in the message queue of the target topic and a consumption progress of the data consumer for the data in the message queue of the target topic.
The offset information includes an offset of the data in the message queue for at least one statistical period, which can be seen in particular in fig. 2. The consumption schedule includes an offset in the message queue of data that has been consumed by the data consumer. For example, the consumption progress may be an offset of the data currently consumed by the data consumer. The sequence of acquiring the offset information and the consumption progress by the monitoring system 100 is not limited in the present application, and in some implementation manners, the monitoring system 100 may periodically acquire the offset information and then acquire the consumption progress. For example: the monitoring system 100 obtains the offset of the data in the message queue at 1:30, the monitoring system 100 obtains the offset of the data in the message queue at 1:35 … …, and the monitoring system 100 obtains the offset of the data in the message queue at 4: 00. I.e., the monitoring system 100 counts the offset of the data in the message queue every 5 minutes. If the lag time needs to be determined at time 4:03, the monitoring system 100 determines the lag time according to the consumption progress, time 4:00, and the statistical offset of the data in the message queue before time 4: 00.
S304: the monitoring system 100 determines a lag time of the data consumer based on the offset information and the consumption progress.
As described above, the monitoring system 100 counts the offset of the message queue data once every 5 minutes, and then can save the offset for each counting period to form offset information, which is convenient for the subsequent determination of the lag time. Then, when the lag time needs to be determined, the offset information and the consumption progress can be directly called to determine the lag time. For example, using the offset information shown in fig. 2 as an example, if the consumption progress indicates that the offset of the data consumed by the data consumer is 2000, the statistical time of the data may be determined to be 1:30, and if the current time is 4:00, the lag time may be determined to be 150 minutes.
S306: the monitoring system 100 adjusts the data retention time of the message queue based on the lag time.
Due to the cleaning mechanism of kafka itself, when the lag time of the data in the message queue exceeds the data holding time, the data in the message queue is cleaned. In order to avoid data being cleared, the data saving time is adjusted, for example, the data saving time is delayed, before the lag time exceeds the data saving time, and then the situation that the data in the message queue is cleared can be avoided. The monitoring system 100 is thus able to reduce the risk of data loss.
In some implementations, the monitoring system 100 can determine a relationship between a lag time and a lag time threshold, where the lag time threshold can be predetermined and the lag time threshold is less than the data retention time. When the lag time is greater than or equal to the lag time threshold, adjusting the data retention time of the message queue.
In other implementations, the monitoring system 100 can determine the lag time in relation to a remaining time threshold, wherein the remaining time threshold can be predetermined and the remaining time threshold is greater than 0. The data retention time of the message queue is adjusted when the difference between the lag time and the data retention time of the message queue is less than or equal to the remaining time threshold.
The monitoring system 100 may determine different extension times according to different scenarios, the extension time referring to a time added on the basis of the data holding time of the message queue. The monitoring system 100 may further set a maximum data storage time, where the maximum data storage time represents a maximum time that data in the message queue can be stored, and the monitoring system 100 cannot adjust the data storage time any more after the lag time exceeds the maximum data storage time.
In some scenarios, the sum of the lag time and the data holding time of the message queue is greater than or equal to the maximum data holding time of the message queue, and the monitoring system 100 adjusts the data holding time to the maximum data holding time. In other scenarios, where the sum of the lag time and the data retention time of the message queue is less than the maximum retention time of the data of the message queue, the monitoring system 100 adjusts the data retention time to the sum of the data retention time and the lag time. Therefore, the monitoring system 100 can prolong the data storage time of the message queue, and further avoid the clearing mechanism of kafka from clearing the data which is not consumed by the data consumer in the message queue.
The method obtains offset information of data in a message queue of a target theme and consumption progress of a data consumer on the data in the message queue of the target theme. The offset information comprises the offset of the data in the message queue in at least one statistical period, the message progress comprises the offset of the data consumed by the data consumer in the message queue, and the method can determine the lag time of the data consumer according to the offset information and the consumption progress, namely the lag time is the lag time of the data in the message queue consumed by the data consumer and the latest data in the message queue, and then adjust the data storage time of the message queue according to the lag time, so that the situation that the data in the message queue is cleared due to the fact that the data storage time is exceeded is avoided. Therefore, the method can reduce the risk of data loss.
While the monitoring system 100 has been described above as adjusting the data retention time of the message team, in some scenarios the monitoring system 100 may also send alert information to the data consumer. Referring to the flowchart of the method for processing data in the message queue shown in fig. 4, the method further includes, on the basis of S302, S304, and S306:
s308: the monitoring system 100 obtains alarm configuration information for the data consumer.
Different data consumers have different degrees of sensitivity to different alarm types. For example, data consumer A is more sensitive to the lag time of data in the message queue, and data consumer B is more sensitive to the lag data volume of data in the message queue. The monitoring system 100 thus needs to determine different alarm conditions based on the alarm configuration information of the unused data consumers.
S310: the monitoring system 100 generates alarm conditions based on the alarm configuration information.
After the monitoring system 100 obtains the alarm configuration information, an alarm condition can be generated according to the alarm configuration information. For example, if the alarm configuration information indicates that the data consumer is more sensitive to the lag time, a condition relating to the lag time is generated, and if the alarm configuration information indicates that the data consumer is more sensitive to the lag data amount, a condition relating to the lag data amount is generated.
S312: the monitoring system 100 performs an alarm based on the alarm condition.
The alarm condition may be that the lag time is greater than or equal to an alarm time threshold, or that the lag data amount is greater than or equal to an alarm data amount threshold. In some scenarios, if the alarm condition is that the lag time is greater than or equal to the alarm time threshold, the monitoring system 100 feeds back alarm information to the data consumer when the lag time is greater than or equal to the alarm time threshold. In other scenarios, if the alarm condition is that the amount of the lag data is greater than or equal to the alarm data amount threshold, the monitoring system needs to determine the amount of the lag data of the data consumer according to the offset information and the consumption progress, and when the amount of the lag data is greater than or equal to the alarm data amount threshold, the alarm information is fed back to the data consumer. The warning information may include consumption progress and the like.
It should be noted that, in some implementations, the above S308, S310 and S312 may be performed before S304, or may be performed before S306, which is only one schematic flowchart.
In this embodiment, the monitoring system 100 can give an alarm to the data consumer, so that the data consumer can obtain the consumption progress of the data consumer, and accelerate consumption of data in the message queue under the prompt of the alarm information. Therefore, a data consumer does not need to actively send a consumption progress acquisition request to a server corresponding to the kafka to acquire the consumption progress, and the access pressure of the server is further reduced. Further, the monitoring system 100 also supports different alarm conditions generated for different data consumers, and improves the adaptability of the monitoring system 100.
The data processing method in the message queue provided by the embodiment of the present application is described above with reference to fig. 1 to 4, and next, a data processing apparatus in the message queue and a computing device for implementing the function of the data processing apparatus in the message queue provided by the embodiment of the present application are described with reference to the accompanying drawings.
As shown in fig. 5, an embodiment of the present application further provides a device 700 for processing data in a message queue, where the device 700 is configured to execute the method for processing data in a message queue. The embodiment of the present application does not limit the division of the functional modules in the apparatus 700, and the following exemplary provides a division of the functional modules:
the in-message-queue data processing apparatus 700 includes an acquisition module 702, a determination module 704, and an adjustment module 706.
An obtaining module 702, configured to obtain offset information of data in a message queue of a target topic and a consumption progress of a data consumer for the data in the message queue of the target topic, where the offset information of the data includes an offset of the data in the message queue in at least one statistical period, and the consumption progress includes an offset of the data that has been consumed by the data consumer in the message queue. The determining module 704 is configured to determine a lag time of the data consumer according to the offset information and the consumption progress. And an adjusting module 706, configured to adjust the data saving time of the message queue according to the lag time.
The data processing apparatus 700 in the message queue may be implemented by a computing device. Fig. 6 provides a computing device, and as shown in fig. 6, the computing device 800 may be specifically configured to implement the functions of the data processing apparatus 700 in the message queue in the embodiment shown in fig. 5.
Computing device 800 includes a bus 801, a processor 802, and a memory 803. Communication between the processor 802 and the memory 803 is via a bus 801.
The processor 802 may be any one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Micro Processor (MP), a Digital Signal Processor (DSP), and the like.
The memory 803 may include a volatile memory (volatile memory), such as a Random Access Memory (RAM). The memory 803 may also include a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard drive (HDD) or a Solid State Drive (SSD).
The memory 803 stores executable program code that the processor 802 executes to perform the aforementioned message queue data processing method. Specifically, the processor 802 executes the program codes to obtain offset information of data in the message queue of the target topic and a consumption progress of the data consumer for the data in the message queue of the target topic, where the offset information of the data includes an offset of the data in the message queue in at least one statistical period, and the consumption progress includes an offset of the data consumed by the data consumer in the message queue. The processor 802 then determines a lag time of the data consumer based on the offset information and the consumption progress, and adjusts the data holding time of the message queue based on the lag time.
The embodiment of the application also provides a computer readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store or a data storage device, such as a data center, that contains one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk), among others. The computer-readable storage medium includes instructions that direct a computing device to perform the above-described method for processing data in a message queue.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the apparatus embodiment, since it is substantially similar to the method embodiment, it is relatively simple to describe, and reference may be made to some descriptions of the method embodiment for relevant points. The above-described system embodiments are merely illustrative, and the units and modules described as separate components may or may not be physically separate. In addition, some or all of the units and modules may be selected according to actual needs to achieve the purpose of the solution of the embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
It should be understood that in the present application, "at least one" means one or more, "a plurality" means two or more. "and/or" for describing an association relationship of associated objects, indicating that there may be three relationships, e.g., "a and/or B" may indicate: only A, only B and both A and B are present, wherein A and B may be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of single item(s) or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.
The foregoing is merely a preferred embodiment of the present application and is not intended to limit the present application in any way. Although the present application has been described with reference to the preferred embodiments, it is not intended to limit the present application. Those skilled in the art can now make numerous possible variations and modifications to the disclosed embodiments, or modify equivalent embodiments, using the methods and techniques disclosed above, without departing from the scope of the claimed embodiments. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present application still fall within the protection scope of the technical solution of the present application without departing from the content of the technical solution of the present application.
Claims (10)
1. A method for processing data in a message queue, comprising:
acquiring offset information of data in a message queue of a target topic and consumption progress of a data consumer for the data in the message queue of the target topic, wherein the offset information of the data comprises an offset of the data in the message queue in at least one statistical period, and the consumption progress comprises the offset of the data consumed by the data consumer in the message queue;
determining a lag time of the data consumer according to the offset information and the consumption progress;
and adjusting the data storage time of the message queue according to the lag time.
2. The method of claim 1, further comprising: and when the lag time is larger than or equal to an alarm time threshold value, feeding back alarm information to the data consumer.
3. The method of claim 1, further comprising:
determining a lag data amount of the data consumer according to the offset information and the consumption progress;
and when the hysteresis data volume is larger than or equal to the alarm data volume threshold value, feeding back alarm information to the data consumer.
4. A method according to claim 2 or 3, characterized in that the method further comprises:
acquiring alarm configuration information of a data consumer;
and generating an alarm condition according to the alarm configuration information, wherein the alarm condition is used for triggering an alarm process when being met.
5. The method of any of claims 1 to 4, wherein said adjusting the data retention time of the message queue according to the lag time comprises:
when the lag time is greater than or equal to a lag time threshold, adjusting the data retention time of the message queue;
or the like, or, alternatively,
adjusting the data retention time of the message queue when the difference between the lag time and the data retention time of the message queue is less than or equal to a remaining time threshold.
6. The method of any of claims 1 to 5, wherein said adjusting the data retention time of the message queue comprises:
if the sum of the lag time and the data storage time of the message queue is greater than or equal to the maximum data storage time of the message queue, adjusting the data storage time to the maximum data storage time;
and if the sum of the lag time and the data storage time of the message queue is less than the maximum data storage time of the message queue, adjusting the data storage time to the sum of the data storage time and the lag time.
7. An apparatus for processing data in a message queue, comprising: the device comprises an acquisition module, a determination module and an adjustment module;
the obtaining module is configured to obtain offset information of data in a message queue of a target topic and a consumption progress of a data consumer for the data in the message queue of the target topic, where the offset information of the data includes an offset of the data in the message queue in at least one statistical period, and the consumption progress includes an offset of the data consumed by the data consumer in the message queue;
the determining module is used for determining the lag time of the data consumer according to the offset information and the consumption progress;
and the adjusting module is used for adjusting the data storage time of the message queue according to the lag time.
8. The apparatus of claim 7, further comprising: an alarm module;
and the alarm module is used for feeding back alarm information to the data consumer when the lag time is greater than or equal to an alarm time threshold.
9. An apparatus for data processing in a message queue, the apparatus comprising: a memory and a processor;
the memory is used for storing a computer program and transmitting the computer program to the processor;
the processor, according to instructions in the computer program, performing the method of data processing in a message queue of any of claims 1 to 6.
10. A computer-readable storage medium for storing computer software instructions for a data processing system in a message queue, which when run on a computer, cause the computer to perform the method of processing data in a message queue as claimed in any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011494838.2A CN112463416A (en) | 2020-12-17 | 2020-12-17 | Method, device, equipment and storage medium for processing data in message queue |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011494838.2A CN112463416A (en) | 2020-12-17 | 2020-12-17 | Method, device, equipment and storage medium for processing data in message queue |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112463416A true CN112463416A (en) | 2021-03-09 |
Family
ID=74803584
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011494838.2A Pending CN112463416A (en) | 2020-12-17 | 2020-12-17 | Method, device, equipment and storage medium for processing data in message queue |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112463416A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115174682A (en) * | 2022-06-17 | 2022-10-11 | 中国银联股份有限公司 | Data distribution method, system, center, device and computer readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106598760A (en) * | 2016-12-19 | 2017-04-26 | 北京奇虎科技有限公司 | Message queue consumption method and apparatus |
| CN109445965A (en) * | 2018-11-07 | 2019-03-08 | 北京明朝万达科技股份有限公司 | By the Redis and MySQL message treatment method realized and equipment |
| CN110187980A (en) * | 2019-04-19 | 2019-08-30 | 北京三快在线科技有限公司 | Message queue monitoring method, device, electronic equipment and readable storage medium storing program for executing |
| CN110971430A (en) * | 2018-09-29 | 2020-04-07 | 北京国双科技有限公司 | Automatic capacity expansion and reduction control method and device, storage medium and processor |
| CN111324392A (en) * | 2018-12-13 | 2020-06-23 | 北京京东尚科信息技术有限公司 | Method and device for automatically adjusting data processing time window |
-
2020
- 2020-12-17 CN CN202011494838.2A patent/CN112463416A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106598760A (en) * | 2016-12-19 | 2017-04-26 | 北京奇虎科技有限公司 | Message queue consumption method and apparatus |
| CN110971430A (en) * | 2018-09-29 | 2020-04-07 | 北京国双科技有限公司 | Automatic capacity expansion and reduction control method and device, storage medium and processor |
| CN109445965A (en) * | 2018-11-07 | 2019-03-08 | 北京明朝万达科技股份有限公司 | By the Redis and MySQL message treatment method realized and equipment |
| CN111324392A (en) * | 2018-12-13 | 2020-06-23 | 北京京东尚科信息技术有限公司 | Method and device for automatically adjusting data processing time window |
| CN110187980A (en) * | 2019-04-19 | 2019-08-30 | 北京三快在线科技有限公司 | Message queue monitoring method, device, electronic equipment and readable storage medium storing program for executing |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115174682A (en) * | 2022-06-17 | 2022-10-11 | 中国银联股份有限公司 | Data distribution method, system, center, device and computer readable storage medium |
| CN115174682B (en) * | 2022-06-17 | 2023-07-28 | 中国银联股份有限公司 | Data distribution method, system, center, device and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109347757B (en) | Message congestion control method, system, device and storage medium | |
| US20190036762A1 (en) | Exception monitoring and alarming method and apparatus | |
| CN112286672B (en) | Method, computing device, and computer storage medium for scheduling tasks | |
| US9110745B2 (en) | System and method for flow control in a messaging subsystem based on message-in/out rates | |
| CN109240836B (en) | Method and device for configuring messages of message queue | |
| CN109560976B (en) | Message delay monitoring method and device | |
| CN109597800B (en) | Log distribution method and device | |
| CN111611020A (en) | Application process searching and killing method and device for application program | |
| CN111625456A (en) | Stuck positioning method and device | |
| CN112463416A (en) | Method, device, equipment and storage medium for processing data in message queue | |
| US20150286548A1 (en) | Information processing device and method | |
| US20210281645A1 (en) | Execution control method and information processing apparatus | |
| US8976803B2 (en) | Monitoring resource congestion in a network processor | |
| CN110347518B (en) | Message processing method and device | |
| CN107025148B (en) | Mass data processing method and device | |
| CN114461323B (en) | Clamping and processing method and device, electronic equipment and storage medium | |
| CN115686367A (en) | Storage space prediction method and device and computer readable storage medium | |
| CN105264499B (en) | Message treatment method, device and reception core in a kind of shared queue | |
| CN113709214A (en) | Message processing method and device, electronic equipment and storage medium | |
| JP2002366393A (en) | Method for collecting computer operation information, implementing system therefor, and its processing program | |
| CN112583742A (en) | Message processing method and device | |
| CN113127221B (en) | Method, device, equipment and storage medium for limiting message consumption rate | |
| CN115174289B (en) | Log data storage method and device, storage medium and electronic device | |
| CN113010366B (en) | Log output method, device and equipment | |
| JP4758412B2 (en) | Distributed resource acquisition scheduling system and record acquisition program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |