CN111600774A - Consumption delay determination method, system, device, equipment and readable storage medium - Google Patents
Consumption delay determination method, system, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN111600774A CN111600774A CN202010401680.3A CN202010401680A CN111600774A CN 111600774 A CN111600774 A CN 111600774A CN 202010401680 A CN202010401680 A CN 202010401680A CN 111600774 A CN111600774 A CN 111600774A
- Authority
- CN
- China
- Prior art keywords
- message
- consumption
- partition
- time information
- delay
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0852—Delays
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
- G06F16/24556—Aggregation; Duplicate elimination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Environmental & Geological Engineering (AREA)
- Computational Linguistics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides a method, a system, a device, equipment and a readable storage medium for determining consumption delay based on a distributed message system, wherein the method comprises the following steps: acquiring the message position which can be consumed by a consumption group in a partition of a message channel and the actual consumed message position of the consumption group in the partition; determining first time information corresponding to the first message and second time information corresponding to the second message, wherein the first message is a message corresponding to a message position which can be consumed by a consumption group, the second message is a message corresponding to a message position which is actually consumed by the consumption group, and the first time information and the second time information represent the same type of time; and calculating a difference value between the first time information and the second time information, and using the difference value as consumption delay information for the partition. In this way, the accuracy of the delay situation for consumption can be improved.
Description
Technical Field
The present invention relates to the field of computer application technologies, and in particular, to a method, a system, an apparatus, a device, and a readable storage medium for determining consumption delay based on a distributed message system.
Background
The distributed message system has wide application, can realize the production and consumption process aiming at the messages, and can intuitively reflect the consumption condition of a consumption group aiming at the consumption progress of the messages. In the actual application process, the consumption group can backlog the consumable messages due to the factors such as the performance of the consumption group, and the consumption of the consumption group can be understood to be delayed. In the prior art, the number of messages actually consumed by the consumption group lags behind the number of messages that can be consumed by the consumption group, and the backlog of consumption can also be understood as the backlog number of messages.
However, due to factors such as different message writing rates of different partitions, the consumed backlog situation is represented by the backlog message number, and the determined consumed delay situation is less accurate. For example, for a partition with a message write rate QPS of ten thousand levels, the number of messages backlogged in 5 seconds is 500 ten thousand, and if the consuming capability of the consuming group is strong, the messages backlogged in the partition can be consumed by the consuming group quickly; the consumption speed is influenced by the QPS, even though the consumption capacity of the consumption group is strong, the number of backlogged messages may reach 500 ten thousand in one day for the partition with the QPS number level, and thus the consumption group may take a long time to complete the consumption. It can be seen that the number of backlogged messages reaches 500 ten thousand, but the reflected severity of the backlog is completely different, that is, the consumed delay condition cannot be accurately reflected by the number of backlogged messages, that is, the accuracy of the determined consumed delay condition is low.
Disclosure of Invention
Embodiments of the present invention provide a method, a system, an apparatus, a device, and a readable storage medium for determining consumption delay based on a distributed message system, so as to improve accuracy of a delay situation for consumption. The specific technical scheme is as follows:
in a first aspect of the present invention, there is provided a method for determining consumption delay based on a distributed message system, including:
acquiring message positions which can be consumed by a consumption group in a partition of a message channel and message positions which are actually consumed by the consumption group in the partition, wherein the message positions which can be consumed by the consumption group are the latest message positions which are exposed to the consumption group by the partition, and the message positions which are actually consumed by the consumption group are synchronous with or behind the message positions which can be consumed by the consumption group;
determining first time information corresponding to a first message and second time information corresponding to a second message, wherein the first message is a message corresponding to a message position which can be consumed by the consumption group, the second message is a message corresponding to a message position which is actually consumed by the consumption group, and the first time information and the second time information represent the same type of time;
calculating a difference value of the first time information and the second time information, and regarding the difference value as consumption delay information for the partition.
Optionally, after the calculating a difference value between the first time information and the second time information and taking the difference value as consumption delay information for the partition, the method further includes:
and performing aggregation operation on the consumption delay information of each partition in the message channel, and taking a result obtained by the aggregation operation as the consumption delay information aiming at the message channel.
Optionally, the performing aggregation operation on the consumption delay information of each partition in the message channel, and taking a result obtained by the aggregation operation as the consumption delay information for the message channel includes:
calculating the maximum value of the consumption delay information of each partition in the message channel, taking the maximum value as a result obtained by aggregation operation, calculating the average value of the consumption delay information of each partition in the message channel, taking the average value as a result obtained by aggregation operation, and/or calculating the quantile of the consumption delay information of each partition in the message channel, and taking the quantile as a result obtained by aggregation operation.
Optionally, after the calculating a difference value between the first time information and the second time information and taking the difference value as consumption delay information for the partition, the method further includes:
for each partition in the message channel, sending consumption delay information of the partition to a data aggregation system with an aggregation function;
and performing aggregation operation on the consumption delay information of each partition in the message channel through the data aggregation system with the aggregation function, and taking a result obtained by the aggregation operation as delay information aiming at the message channel.
Optionally, after performing aggregation operation on the consumption delay information of each partition in the message channel and taking a result obtained by the aggregation operation as the consumption delay information for the message channel, the method includes:
acquiring a delay threshold corresponding to a service system;
comparing the consumption delay information aiming at the message channel with the delay threshold value, and triggering an alarm aiming at the service system according to the comparison result.
In a second aspect of the present invention, there is also provided a consumption delay determination system based on a distributed message system, including: the system comprises a computing node and a data aggregation system with an aggregation function; the computing nodes comprise a plurality of computing sub-nodes, and each partition of the message channel corresponds to different computing sub-nodes respectively;
aiming at each partition in a message channel, acquiring a message position which can be consumed by a consumption group in the partition and a message position which is actually consumed by the consumption group in the partition by a computing child node corresponding to the partition, wherein the message position which can be consumed by the consumption group is the latest message position which is exposed to the consumption group by the partition, and the message position which is actually consumed by the consumption group is synchronous with or lags behind the message position which can be consumed by the consumption group; determining first time information corresponding to a first message and second time information corresponding to a second message, wherein the first message is a message corresponding to a message position which can be consumed by the consumption group, the second message is a message corresponding to a message position which is actually consumed by the consumption group, and the first time information and the second time information represent the same type of time; calculating a difference value of the first time information and the second time information, and taking the difference value as consumption delay information for the partition; sending the consumption delay information of the partitions to the data aggregation system with the aggregation function;
the data aggregation system with the aggregation function receives consumption delay information of the partitions sent by the computing nodes respectively corresponding to the partitions; and performing aggregation operation on the consumption delay information of each partition in the message channel, and taking a result obtained by the aggregation operation as the consumption delay information aiming at the message channel.
Optionally, the system further includes: a monitoring alarm module;
the data aggregation system with the aggregation function is further used for delivering the consumption delay information of the message channel to the monitoring alarm system;
the monitoring alarm system is used for acquiring a delay threshold corresponding to the service system; comparing the consumption delay information aiming at the message channel with the delay threshold value, and triggering an alarm aiming at the service system according to the comparison result.
In a third aspect of the present invention, there is also provided a consumption delay determining apparatus based on a distributed message system, including:
the first acquisition module is used for acquiring the message positions which can be consumed by a consumption group in a partition of a message channel and the message positions which are actually consumed by the consumption group in the partition, wherein the message positions which can be consumed by the consumption group are the latest message positions which are exposed to the consumption group by the partition, and the message positions which are actually consumed by the consumption group are synchronous with or behind the message positions which can be consumed by the consumption group;
a determining module, configured to determine first time information corresponding to a first message and second time information corresponding to a second message, where the first message is a message corresponding to a message position that can be consumed by the consumption group, the second message is a message corresponding to a message position that is actually consumed by the consumption group, and the first time information and the second time information represent the same type of time;
and the calculating module is used for calculating the difference value of the first time information and the second time information and taking the difference value as consumption delay information aiming at the subareas.
In a fourth aspect of the present invention, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus;
a memory for storing a computer program;
a processor configured to implement the method steps of the first aspect when executing the program stored in the memory.
In yet another aspect of the present invention, there is also provided a computer-readable storage medium having stored therein instructions which, when run on a computer, cause the computer to perform the method steps of the first aspect described above.
In a further aspect of the present invention, there is also provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform the method steps of the first aspect described above.
The consumption delay determining method, the system, the device, the equipment and the readable storage medium based on the distributed message system, provided by the embodiment of the invention, are used for acquiring the message position which can be consumed by the consumption group in the partition of the message channel and the message position which is actually consumed by the consumption group in the partition, wherein the message position which can be consumed by the consumption group is the latest message position which is exposed to the consumption group by the partition, and the message position which is actually consumed by the consumption group is synchronous with or lags behind the message position which can be consumed by the consumption group; determining first time information corresponding to the first message and second time information corresponding to the second message, wherein the first message is a message corresponding to a message position which can be consumed by a consumption group, the second message is a message corresponding to a message position which is actually consumed by the consumption group, and the first time information and the second time information represent the same type of time; and calculating a difference value between the first time information and the second time information, and using the difference value as consumption delay information for the partition. In the embodiment of the invention, the time information of the message corresponding to the message position which can be consumed by the consumption group and the time information of the message corresponding to the message position which is actually consumed by the consumption group are determined, the consumption delay information aiming at the subarea is represented by the difference value of the time information, and the consumption delay information represented by the difference value of the time information can be understood as the time length of the consumption delay, namely the delay, so that the delay condition aiming at consumption can be reflected by the delay condition of consumption, and the accuracy of the delay condition aiming at consumption can be improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below.
Fig. 1 is a flowchart of a consumption delay determination method based on a distributed message system according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a partition message location in accordance with an embodiment of the present invention;
fig. 3 is another flowchart of a consumption delay determination method based on a distributed message system according to an embodiment of the present invention;
fig. 4 is another flowchart of a consumption delay determination method based on a distributed message system according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a consumption delay determining system based on a distributed message system according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a consumption delay determining system based on a distributed message system according to an embodiment of the present invention;
fig. 7 is another flowchart of a consumption delay determination method based on a distributed message system according to an embodiment of the present invention;
FIG. 8A is a diagram illustrating the effect of a delay in consumption by the backlog number of consumption by a partition;
FIG. 8B is a diagram illustrating the effect of message latency in the embodiment of the present invention;
fig. 9A is a schematic structural diagram of a consumption delay determining apparatus based on a distributed message system according to an embodiment of the present invention;
fig. 9B is a schematic structural diagram of a consumption delay determining apparatus based on a distributed message system according to an embodiment of the present invention;
fig. 9C is a schematic structural diagram of a consumption delay determining apparatus based on a distributed message system according to an embodiment of the present invention;
fig. 10 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.
In order to improve the accuracy of a delay situation for consumption, the embodiment of the invention provides a consumption delay determination method based on a distributed message system.
In addition, in the prior art, when the severity of delay is estimated or an alarm is triggered, generally, service personnel and the like estimate whether the delay condition reflected by the overstocked message quantity is serious according to experience values, so that the randomness of determining the delay condition of consumption is strong, the randomness is not objective, and the rationality is poor. The consumption delay determining method based on the distributed message system provided by the embodiment of the invention can also enable the determined delay condition aiming at consumption to be more objective and reasonable. The consumption delay determination method based on the distributed message system provided by the embodiment of the invention is explained in detail below.
The consumption delay determination method based on the distributed message system provided by the embodiment of the invention can be applied to electronic equipment, and specifically, the electronic equipment can be a processor, a terminal and the like.
An embodiment of the present invention provides a method for determining consumption delay based on a distributed message system, as shown in fig. 1, the method may include:
s101, acquiring the message position which can be consumed by a consumption group in a partition of a message channel and the message position which is actually consumed by the consumption group in the partition, wherein the message position which can be consumed by the consumption group is the latest message position which is exposed to the consumption group by the partition, and the message position which is actually consumed by the consumption group is synchronous with or lags behind the message position which can be consumed by the consumption group;
s102, determining first time information corresponding to a first message and second time information corresponding to a second message, wherein the first message is a message corresponding to a message position which can be consumed by a consumption group, the second message is a message corresponding to a message position which is actually consumed by the consumption group, and the first time information and the second time information represent the same type of time;
and S103, calculating a difference value between the first time information and the second time information, and taking the difference value as consumption delay information for the partition.
In the embodiment of the invention, the time information of the message corresponding to the message position which can be consumed by the consumption group and the time information of the message corresponding to the message position which is actually consumed by the consumption group are determined, the consumption delay information aiming at the subarea is represented by the difference value of the time information, and the consumption delay information represented by the difference value of the time information can be understood as the time length of the consumption delay, namely the delay, so that the delay condition aiming at consumption can be reflected by the delay condition of consumption, and the accuracy of the delay condition aiming at consumption can be improved.
Referring to fig. 1, a detailed description is given of a consumption delay determination method based on a distributed message system according to an embodiment of the present invention, which specifically includes:
s101, acquiring the message position which can be consumed by the consumption group in the partition of the message channel and the message position which is actually consumed by the consumption group in the partition.
The message positions that can be consumed by the consumption group are the latest message positions that the partition exposes to the consumption group, and the message positions that are actually consumed by the consumption group are synchronous with or behind the message positions that can be consumed by the consumption group.
The message channels are obtained by dividing according to the message categories stored in the distributed message system.
In the distributed message system, in order to achieve high throughput and high availability, a message service may be provided in a partition manner, specifically, each partition stores messages, and the messages of a plurality of partitions constitute a total message amount.
Specifically, in a distributed message system, such as Kafka (Kafka), Topic is treated as a complete whole by partitioning into smaller units during message sending and storing, where Topic may also be understood as a message channel and is obtained by dividing according to the type of a message stored in the distributed message system. It is understood that in Kafka, Topic is a logical concept of storing messages, and can be considered as a message set. Each message sent to the Kafka cluster has a category, and each Topic may be partitioned into multiple partitions (at least one partition per Topic). Each partition in the Topic receives messages independently, and the messages of each partition are combined together to form the total message amount of the Topic, as shown in fig. 2, a partition in the Topic includes 10 messages stored at message positions 1 to 10, Offset represents the position of a message, such as message positions 1 to 10, and Offset is a monotonically increasing Long integer (Long) in one partition.
The message position shown in position 1 in fig. 2 is LogStartOffset, which indicates the message position of the first message in the partition. The message position shown in position 2 is ConsumerOffset, which represents the position of the message actually consumed by the consuming group. The message location shown in location 3 is HW Offset (Highwatermark), which represents the message location that the consuming group can consume, and can also be understood as the latest location exposed to consumption by the consuming group. The message position shown in position 4 is leo (logendoffset), and indicates position information where the latest message stored in the partition deviates from LogStartOffset. And for one partition, ConsumerOffset < ═ HWOffset < ═ LEO.
In the embodiment of the present invention, a query interface may be obtained by encapsulation, and for each Topic, the acquisition module queries ConsumerOffset and HW Offset in each partition in the Topic through the query interface, that is, obtains the message position actually consumed by the consumption group and the message position that can be consumed by the consumption group.
S102, determining first time information corresponding to the first message and second time information corresponding to the second message.
The first message is a message corresponding to the message position which can be consumed by the consumption group, and the second message is a message corresponding to the message position which is actually consumed by the consumption group.
Each Offset, i.e. message position, can locate a unique message, each message has corresponding time information, and it can also be understood that a message has a unique timestamp. The time information corresponding to each message may be a client creation time of the message or a time when the message falls into a server, specifically, a time when a producer creates the message, may also be understood as a creation time of the message, a time when a message system stores the message into a partition, may also be understood as a storage time of the message, or a time when a consumer consumes the message, or may also be understood as a consumption time of the message, or the like.
Specifically, the production end sends the message to the message system, and simultaneously, the production end can send the timestamp of the message created by the production end to the message system, so that the message system can store the timestamp of the message created while storing the message; or, when the message system stores the message to the message location of the partition, the timestamp stored to the message location may be recorded; alternatively, when a group consumes a message at a message location, a timestamp of the consumption may be recorded. Thus, after the message location of the message is determined, the time information of the message stored in the message location, such as the timestamp of the message, can be obtained. As for the partition in Topic, the timestamp corresponding to HW Offset and the timestamp corresponding to ConsumerOffset may be obtained.
The first time information and the second time information represent the same type of time.
For example, the first time information is a creation time of a message corresponding to a message location that the consuming group can consume, and the second time information is a creation time of a message corresponding to a message location that the consuming group actually consumes. Or the first time information is the storage time of the message corresponding to the message position which can be consumed by the consumption group, and the second time information is the storage time of the message corresponding to the message position which is actually consumed by the consumption group. Or, the first time information is the consumption time of the message corresponding to the message position that can be consumed by the consumption group, and the second time information is the consumption time of the message corresponding to the message position that is actually consumed by the consumption group.
And S103, calculating a difference value between the first time information and the second time information, and taking the difference value as consumption delay information for the partition.
It is simply understood that the time corresponding to the message position where the message actually consumed by the consumption group is located is calculated to be later than the time corresponding to the message position where the message capable of being consumed by the consumption group is located. For example, for a partition in Topic, a difference between the timestamp corresponding to HWOffset and the timestamp corresponding to ConsumerOffset may be calculated, and for example, the difference between the first time information and the second time information is denoted as timestamp (lag), timestamp (lag) ts (HW Offset) -ts (ConsumerOffset), ts (HW Offset) is the timestamp corresponding to HW Offset, and ts (ConsumerOffset) is the timestamp corresponding to ConsumerOffset.
In one implementation manner, the electronic device may include a calculation module, where the calculation module determines first time information corresponding to the first message and second time information corresponding to the second message, and calculates a difference between the first time information and the second time information.
In the embodiment of the invention, the message position which can be consumed by the consumption group in the subarea of the message channel and the message position which is actually consumed by the consumption group in the subarea are obtained, the message position is correspondingly converted into the time information, namely, the time information of the message which corresponds to the message position which can be consumed by the consumption group and the time information of the message which corresponds to the message position which is actually consumed by the consumption group are determined, and the consumption delay information aiming at the subarea is represented by the difference value of the time information, namely, the delay condition aiming at consumption is reflected by the consumption delay, so that the accuracy of the delay condition aiming at consumption can be improved. And the consumption delay information of the subareas is reflected by the difference value between the time information of the message corresponding to the message position which can be consumed by the consumption group and the time information of the message corresponding to the message position which is actually consumed by the consumption group, so that the time after consumption can be represented, and the standardized consumption delay can be reflected.
In an alternative embodiment, after calculating a difference value between the first time information and the second time information and using the difference value as the consumption delay information for the partition, as shown in fig. 3, the method may further include:
and S104, performing aggregation operation on the consumption delay information of each partition in the message channel, and taking the result obtained by the aggregation operation as the consumption delay information aiming at the message channel.
The maximum value of the consumption delay information of each partition in the message channel can be calculated, the maximum value is used as a result obtained by aggregation operation, the average value of the consumption delay information of each partition in the message channel is calculated, the average value is used as a result obtained by aggregation operation, and/or the quantile of the consumption delay information of each partition in the message channel is calculated, and the quantile is used as a result obtained by aggregation operation.
The consumption delay information may be understood as a duration of the consumption delay.
In an alternative embodiment, the maximum value of the consumption delay information of each partition in the message channel may be calculated, and the maximum value is used as the result of the aggregation operation.
Or, in another alternative embodiment, an average value of the consumption delay information of each partition in the message channel may be calculated, and the average value may be used as the result obtained by the aggregation operation.
Or, in another alternative embodiment, the quantiles of the consumption delay information of each partition in the message channel may be calculated, and the quantiles may be used as the result obtained by the aggregation operation.
Alternatively, the maximum value of the consumption delay information of each partition in the message channel, the average value of the consumption delay information of each partition in the message channel, and the quantile of the consumption delay information of each partition in the message channel may be calculated at the same time.
For example, for a consuming group, the consuming group consumes messages in a message channel, such as Topic, which includes N partitions. The consumption delay information of the partition in the above embodiment is a difference between the time corresponding to the message position where the message actually consumed by the consumption group is located and the time corresponding to the message position where the message capable of being consumed by the consumption group is located, and it is understood that the time corresponding to the message position where the message actually consumed by the consumption group is located is a time length behind the time corresponding to the message position where the message capable of being consumed by the consumption group is located. The consumption delay information of the partition is represented by a partition TS.
The maximum value of the consumption delay information of each partition in the message channel, such as Max Timestamp, may be calculated by using a formula Max Timestamp ═ Max (partition ts).
The average value of the consumption delay information of each partition in the message channel, such as Avg Timestamp, may be calculated by a formula Avg Timestamp ═ sum (partition ts)/N.
The quantile of the consumption delay information of each partition in the message channel is calculated, for example, P99 Timestamp is calculated according to the formula P99 Timestamp ═ P99(partition TS).
In one implementation, P99 may be calculated, which may be understood as a quantile with a probability of 99%, and in particular, a value reached by 99% of the duration of the consumption delay for each partition in the message channel. In addition, the quantile with the probability of other values, such as 80%, 90%, etc., may be calculated.
The time information corresponding to the message position is analyzed by performing secondary consumption aiming at Offset of the message channel in the distributed message system, specifically, the time information can be a timestamp corresponding to the message position, and the timestamps of all the partitions of the message channel are aggregated and calculated aiming at one message channel, so that the consumption delay monitoring function in a brand-new true sense can be realized, compared with the prior art that the consumption delay is subjectively monitored by Offset, the consumption delay condition can be objectively and accurately reflected.
In an alternative embodiment, at S103: after calculating the difference between the first time information and the second time information and using the difference as the consumption delay information for the partition, as shown in fig. 4, the method may further include:
s401, aiming at each partition in the message channel, the consumption delay information of the partition is sent to a data aggregation system with an aggregation function.
S402, carrying out aggregation operation on the consumption delay information of each partition in the message channel through a data aggregation system with an aggregation function, and taking the result obtained by the aggregation operation as the delay information aiming at the message channel.
The consumption delay information of different partitions can be calculated according to the different partitions.
In one implementation, the consumption delay information of different partitions can be calculated by a plurality of calculation child nodes respectively. The partitions may include identification information of T1 CG 1P 1, T1 CG 1P 2, T1 CG 1P 23, T-nCG-nPnEtc. Specifically, one computation sub-node corresponds to one partition, and the computation sub-node may obtain, from the online cluster, a message position that can be consumed by the consumption group in the partition and a message position actually consumed by the consumption group in the partition, where the message position may be represented by an Offset of the message, and calculate consumption delay information of the partition. Each calculation sub-node does not influence, and each calculation sub-node only calculates the consumption delay information of the corresponding partition. At the moment, the computing sub-nodes are stateless, the distributed deployment characteristics can be fully utilized to deploy each computing sub-node, distributed computing is achieved, and compared with the method that consumption delay information of a plurality of partitions is computed in sequence, computing efficiency can be improved.
In an alternative embodiment, the data query system with aggregation function is a database Druid. After each computing node respectively obtains the consumption delay information of the corresponding partition, the consumption delay information of the partition is delivered to the Druid, and the Druid is used as a database, so that the aggregation performance is high, and the aggregation efficiency is high. With the aggregation performance of the draid, fast aggregation is achieved, and thus, the efficiency of determining delay information of a message channel can be improved.
In the embodiment of the invention, the delay information of the message channel can be split according to the partitions, for example, after each computing node respectively obtains the consumption delay information of the corresponding partition, the consumption delay information of the partition is delivered to the Druid, the general Druid realizes aggregation aiming at consumption delay, the aggregation performance is improved according to the order of magnitude, and the rapid aggregation is realized.
If the consumption delay information of each partition is calculated in sequence and aggregation is realized by the calculation module for calculating the consumption delay information of the partitions, the consumption delay information of each partition is aggregated to obtain the delay information of the message channel, and a large amount of time may be consumed for data acquisition and aggregation. Because aggregation operation needs to acquire all data at the same position, that is, consumption delay information of each partition, a cluster may have hundreds of topics, a Topic may have hundreds of partitions, and a Topic may have tens of consumption groups for consumption, at this time, if all the calculation is completed by one calculation module, the calculation amount required for one Topic may be millions of orders of magnitude, and a single calculation module cannot be completed at all or has low calculation efficiency.
An embodiment of the present invention provides a consumption delay determining system based on a distributed message system, as shown in fig. 5, the consumption delay determining system may include: a computing node 501 and a data aggregation system 502 with aggregation function; the computing node 501 includes a plurality of computing sub-nodes, and each partition of the message channel corresponds to a different computing sub-node.
Aiming at each partition in a message channel, acquiring a message position which can be consumed by a consumption group in the partition and a message position which is actually consumed by the consumption group in the partition by a computing child node corresponding to the partition, wherein the message position which can be consumed by the consumption group is the latest message position which is exposed to the consumption group by the partition, and the message position which is actually consumed by the consumption group is synchronous with or lags behind the message position which can be consumed by the consumption group; determining first time information corresponding to the first message and second time information corresponding to the second message, wherein the first message is a message corresponding to a message position which can be consumed by a consumption group, the second message is a message corresponding to a message position which is actually consumed by the consumption group, and the first time information and the second time information represent the same type of time; calculating a difference value between the first time information and the second time information, and taking the difference value as consumption delay information for the partition; the consumption delay information of the partition is sent to the data aggregation system 502 having an aggregation function.
The data aggregation system 502 with an aggregation function receives consumption delay information of partitions sent by computing nodes respectively corresponding to the partitions; and performing aggregation operation on the consumption delay information of each partition in the message channel, and taking the result obtained by the aggregation operation as the consumption delay information aiming at the message channel.
In the embodiment of the invention, each computing node only computes the consumption delay information of the corresponding partition. And after obtaining the consumption delay information of the corresponding partitions, each computing node delivers the consumption delay information of the partitions to a data query system with an aggregation function. And aggregating by using an aggregation function of the data query system, namely aggregating the consumption delay information of each partition by using the data query system. Therefore, the consumption delay information of the calculation partition can be separated from the aggregation operation, and the aggregation function of the data query system is utilized to improve the calculation efficiency.
In an optional embodiment, as shown in fig. 6, the consumption delay determination system based on a distributed message system according to an embodiment of the present invention may further include: a monitoring alarm module 503.
The data aggregation system 502 with the aggregation function is also used for delivering the consumption delay information of the message channel to the monitoring alarm system 503;
the monitoring alarm system 503 is configured to obtain a delay threshold corresponding to the service system; comparing the consumption delay information for the message channel with a delay threshold, and triggering an alarm for the service system according to the comparison result.
In the embodiment of the invention, after each computing node respectively obtains the consumption delay information of the corresponding subarea, the consumption delay information of the subarea is delivered to the data query system with the aggregation function, the consumption delay information of each subarea is aggregated through the data query system, and the consumption delay information of the aggregated message channel is delivered to the monitoring alarm system 503, so that the alarm is triggered through the monitoring alarm system, the calculation efficiency of calculating the consumption delay information of the message channel is improved, the timeliness of triggering the alarm according to the consumption delay information of the message channel can be further improved, namely the timeliness of monitoring the consumption delay is improved.
The embodiment of the invention can be applied to systems and the like which pay attention to the consumption delay and have strict requirements on monitoring timeliness, and the monitoring timeliness is improved. The quality change of the service consumption delay monitoring use mode can be realized, and the alarm subscription can support the accurate customization of the second level. In a specific example, the method is applied to advertisement miraculous services, aiming at alarm monitoring which is very dependent in downstream real-time link monitoring, the monitoring timeliness can be improved from a small time level to a second level according to the test effect.
In an alternative embodiment, at S104: after performing aggregation operation on the consumption delay information of each partition in the message channel and taking the result obtained by the aggregation operation as the consumption delay information for the message channel, as shown in fig. 7, the method may further include:
and S105, acquiring a delay threshold corresponding to the service system.
The delay threshold may be a threshold determined according to traffic conditions.
The larger the difference between the first time information and the second time information, that is, the larger the difference between the time information of the message corresponding to the message position that can be consumed by the consuming group and the time information of the message corresponding to the message position that is actually consumed by the consuming group, the more serious the consumption delay is.
S106, comparing the consumption delay information aiming at the message channel with a delay threshold value, and triggering an alarm aiming at the service system according to the comparison result.
In one implementation, the alarm may be triggered when the delay information of the message channel is greater than or equal to a delay threshold.
In an implementation manner, the obtained delay information of the message channel can be sent to a monitoring system, the delay condition of the message channel is monitored by the monitoring system, and an alarm can be triggered when the consumption delay information of the message channel is greater than or equal to a delay threshold value.
In an implementation manner, the monitoring system can set different delay thresholds for different services, so that an alarm can be triggered according to actual requirements of different services, and monitoring of consumption delay is realized.
In an alternative embodiment, the electronic device may include a monitoring alarm module, through which monitoring and alarming for delayed situations of consumption is achieved.
In the embodiment of the invention, the consumption delay condition is reflected by the difference value between the time information of the message corresponding to the message position which can be consumed by the consumption group and the time information of the message corresponding to the message position which is actually consumed by the consumption group, so that the consumption delay can be monitored.
In the prior art, the delay of consumption is represented by the backlog number of consumption of a partition, specifically, for one Topic, ConsumerOffset and HW Offset in each partition in the Topic, that is, the message position actually consumed by a consumption group and the message position capable of being consumed by the consumption group are obtained, and then for each partition, the difference between HW Offset and Consumer Offset is calculated, and the difference between HW Offset and Consumer Offset represents the lagging condition of the consumption group of the partition, and is represented by lags. The sum of the lags of all the partitions represents the delay of a consumer group in a consumer group of a Topic, and as shown in fig. 8A, it can show the variation of the backlogged number of messages in different time ranges, where the unit of the backlogged number of messages is ten thousand, and as the Lag at a certain time is 1000 ten thousand, it represents that the consumer group consumes 1000 ten thousand messages later. The number of messages backlogged in different time frames may be presented, such as 3 hours, 1 day, 3 days, 7 days, or 1 month, etc.
In the embodiment of the invention, the Topic/consumer group can be used as a dimension to perform aggregation calculation, the delay condition of a consumption group in one Topic can be obtained, the result obtained by the aggregation calculation can be used as delay information aiming at a message channel, the message channel is delivered to a monitoring system, the monitoring system can open a self-defined interface, and a service can set respective delay threshold value and give an alarm in a self-defined manner.
In a specific example, for a consumer group: t.qx.tracking.feature.join.retry at Topic: the delay consumed by t _ qx _ retry, i.e. the delay duration, can show the variation of the delay duration in different time ranges, as shown in fig. 8B. The time delay duration in different time ranges, such as 3 hours, 1 day, 3 days, 7 days, or 1 month, can be displayed in the embodiment of the present invention. In the embodiment of the present invention, the maximum value of the consumption delay information of each partition in Topic may be shown, where the maximum value is the consumption delay information of Topic. It is also possible to show the average value of the consumption delay information of each partition in Topic, which is the consumption delay information of Topic. And displaying the P99 value calculated by the consumption delay information of each partition in the Topic, wherein the P99 value is the consumption delay information of the Topic. Or simultaneously displaying the maximum value of the consumption delay information of each partition in the Topic, the average value of the consumption delay information of each partition in the Topic and the P99 value calculated by the consumption delay information of each partition in the Topic.
In contrast, in the embodiment of the present invention, aggregation operation may be performed on the consumption delay information obtained from each partition to obtain the consumption delay information of Topic, and a true and standardized consumption delay monitoring may be implemented by the consumption delay duration, that is, how long the consumption lags behind. And particularly, aggregation operation in different modes can be performed, so that different requirements in the practical application process can be met.
Corresponding to the consumption delay determination method based on the distributed message system provided in the foregoing embodiment, an embodiment of the present invention provides a consumption delay determination apparatus based on the distributed message system, as shown in fig. 9A, where the consumption delay determination apparatus may include:
a first obtaining module 901, configured to obtain a message location that can be consumed by a consuming group in a partition of a message channel, and a message location that is actually consumed by the consuming group in the partition, where the message location that can be consumed by the consuming group is a latest message location that the partition exposes to the consuming group, and the message location that is actually consumed by the consuming group is synchronized with or lags behind the message location that can be consumed by the consuming group;
a determining module 902, configured to determine first time information corresponding to a first message and second time information corresponding to a second message, where the first message is a message that can be consumed by a consuming group and corresponds to a message position that is actually consumed by the consuming group, and the first time information and the second time information represent the same type of time;
a calculating module 903, configured to calculate a difference between the first time information and the second time information, and use the difference as consumption delay information for the partition.
In the embodiment of the invention, the time information of the message corresponding to the message position which can be consumed by the consumption group and the time information of the message corresponding to the message position which is actually consumed by the consumption group are determined, the consumption delay information aiming at the subarea is represented by the difference value of the time information, and the consumption delay information represented by the difference value of the time information can be understood as the time length of the consumption delay, namely the delay, so that the delay condition aiming at the consumption can be reflected by the delay of the consumption, and the accuracy of the delay condition aiming at the consumption can be improved.
Optionally, as shown in fig. 9B, the apparatus further includes:
and an aggregation operation module 904, configured to perform an aggregation operation on the consumption delay information of each partition in the message channel after calculating a difference between the first time information and the second time information and taking the difference as consumption delay information for the partition, and taking a result obtained by the aggregation operation as consumption delay information for the message channel.
Optionally, the aggregation operation module 904 is specifically configured to calculate a maximum value of the consumption delay information of each partition in the message channel, use the maximum value as a result obtained through the aggregation operation, calculate an average value of the consumption delay information of each partition in the message channel, use the average value as a result obtained through the aggregation operation, and/or calculate a quantile of the consumption delay information of each partition in the message channel, and use the quantile as a result obtained through the aggregation operation.
Optionally, the aggregation operation module 904 is specifically configured to calculate a difference between the first time information and the second time information, and send consumption delay information of the partition to the data aggregation system with an aggregation function for each partition in the message channel after the difference is used as consumption delay information for the partition; and performing aggregation operation on the consumption delay information of each partition in the message channel through a data aggregation system with an aggregation function, and taking the result obtained by the aggregation operation as the delay information aiming at the message channel.
Optionally, as shown in fig. 9C, the apparatus further includes:
a second obtaining module 905, configured to obtain a delay threshold corresponding to the service system after performing aggregation operation on the consumption delay information of each partition in the message channel and taking a result obtained through the aggregation operation as the consumption delay information for the message channel;
an alarm module 906, configured to compare the consumption delay information for the message channel with a delay threshold, and trigger an alarm for the service system according to a result of the comparison.
The consumption delay determining device based on the distributed message system provided by the embodiment of the invention is a device applying the consumption delay determining method based on the distributed message system, so that all the embodiments of the consumption delay determining method based on the distributed message device are suitable for the device and can achieve the same or similar beneficial effects.
Corresponding to the consumption delay determination method based on the distributed message system provided in the foregoing embodiment, an embodiment of the present invention further provides an electronic device, as shown in fig. 10, including a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004, where the processor 1001, the communication interface 1002, and the memory 1003 complete communication with each other through the communication bus 1004.
A memory 1003 for storing a computer program;
the processor 1001 is configured to implement the method steps of the consumption delay determination method based on the distributed message system according to the foregoing embodiment when executing the program stored in the memory 1003.
In the embodiment of the invention, the time information of the message corresponding to the message position which can be consumed by the consumption group and the time information of the message corresponding to the message position which is actually consumed by the consumption group are determined, the consumption delay information aiming at the subarea is represented by the difference value of the time information, and the consumption delay information represented by the difference value of the time information can be understood as the time length of the consumption delay, namely the delay, so that the delay condition aiming at consumption can be reflected by the delay condition of consumption, and the accuracy of the delay condition aiming at consumption can be improved.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.
In yet another embodiment of the present invention, there is provided a computer-readable storage medium having stored therein instructions, which, when executed on a computer, cause the computer to perform the method steps of the distributed message system-based consumption delay determination method provided by the above-mentioned embodiment.
In the embodiment of the invention, the time information of the message corresponding to the message position which can be consumed by the consumption group and the time information of the message corresponding to the message position which is actually consumed by the consumption group are determined, the consumption delay information aiming at the subarea is represented by the difference value of the time information, and the consumption delay information represented by the difference value of the time information can be understood as the time length of the consumption delay, namely the delay, so that the delay condition aiming at consumption can be reflected by the delay condition of consumption, and the accuracy of the delay condition aiming at consumption can be improved.
In a further embodiment of the present invention, there is provided a computer program product containing instructions, which when run on a computer, causes the computer to perform the method steps of the distributed message system based consumption delay determination method provided by the above embodiment.
In the embodiment of the invention, the time information of the message corresponding to the message position which can be consumed by the consumption group and the time information of the message corresponding to the message position which is actually consumed by the consumption group are determined, the consumption delay information aiming at the subarea is represented by the difference value of the time information, and the consumption delay information represented by the difference value of the time information can be understood as the time length of the consumption delay, namely the delay, so that the delay condition aiming at consumption can be reflected by the delay condition of consumption, and the accuracy of the delay condition aiming at consumption can be improved.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the system, apparatus, electronic device, computer-readable storage medium, and computer program product embodiments are substantially similar to the method embodiments, so that the description is simple, and reference may be made to some descriptions of the method embodiments for relevant points.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (10)
1. A method for determining consumption delay based on a distributed messaging system, comprising:
acquiring message positions which can be consumed by a consumption group in a partition of a message channel and message positions which are actually consumed by the consumption group in the partition, wherein the message positions which can be consumed by the consumption group are the latest message positions which are exposed to the consumption group by the partition, and the message positions which are actually consumed by the consumption group are synchronous with or behind the message positions which can be consumed by the consumption group;
determining first time information corresponding to a first message and second time information corresponding to a second message, wherein the first message is a message corresponding to a message position which can be consumed by the consumption group, the second message is a message corresponding to a message position which is actually consumed by the consumption group, and the first time information and the second time information represent the same type of time;
calculating a difference value of the first time information and the second time information, and regarding the difference value as consumption delay information for the partition.
2. The method of claim 1, wherein after the calculating the difference between the first time information and the second time information and using the difference as consumption delay information for the partition, the method further comprises:
and performing aggregation operation on the consumption delay information of each partition in the message channel, and taking a result obtained by the aggregation operation as the consumption delay information aiming at the message channel.
3. The method according to claim 2, wherein performing an aggregation operation on the consumption delay information of each partition in the message channel, and taking a result of the aggregation operation as the consumption delay information for the message channel comprises:
calculating the maximum value of the consumption delay information of each partition in the message channel, taking the maximum value as a result obtained by aggregation operation, calculating the average value of the consumption delay information of each partition in the message channel, taking the average value as a result obtained by aggregation operation, and/or calculating the quantile of the consumption delay information of each partition in the message channel, and taking the quantile as a result obtained by aggregation operation.
4. The method of claim 1, wherein after the calculating the difference between the first time information and the second time information and using the difference as consumption delay information for the partition, the method further comprises:
for each partition in the message channel, sending consumption delay information of the partition to a data aggregation system with an aggregation function;
and performing aggregation operation on the consumption delay information of each partition in the message channel through the data aggregation system with the aggregation function, and taking a result obtained by the aggregation operation as delay information aiming at the message channel.
5. The method according to claim 2, wherein after performing an aggregation operation on the consumption delay information of each partition in the message channel and taking a result of the aggregation operation as the consumption delay information for the message channel, the method comprises:
acquiring a delay threshold corresponding to a service system;
comparing the consumption delay information aiming at the message channel with the delay threshold value, and triggering an alarm aiming at the service system according to the comparison result.
6. A system for determining consumption delay based on a distributed messaging system, comprising: the system comprises a computing node and a data aggregation system with an aggregation function; the computing nodes comprise a plurality of computing sub-nodes, and each partition of the message channel corresponds to different computing sub-nodes respectively;
aiming at each partition in a message channel, acquiring a message position which can be consumed by a consumption group in the partition and a message position which is actually consumed by the consumption group in the partition by a computing child node corresponding to the partition, wherein the message position which can be consumed by the consumption group is the latest message position which is exposed to the consumption group by the partition, and the message position which is actually consumed by the consumption group is synchronous with or lags behind the message position which can be consumed by the consumption group; determining first time information corresponding to a first message and second time information corresponding to a second message, wherein the first message is a message corresponding to a message position which can be consumed by the consumption group, the second message is a message corresponding to a message position which is actually consumed by the consumption group, and the first time information and the second time information represent the same type of time; calculating a difference value of the first time information and the second time information, and taking the difference value as consumption delay information for the partition; sending the consumption delay information of the partitions to the data aggregation system with the aggregation function;
the data aggregation system with the aggregation function receives consumption delay information of the partitions sent by the computing nodes respectively corresponding to the partitions; and performing aggregation operation on the consumption delay information of each partition in the message channel, and taking a result obtained by the aggregation operation as the consumption delay information aiming at the message channel.
7. The system of claim 6, further comprising: a monitoring alarm module;
the data aggregation system with the aggregation function is further used for delivering the consumption delay information of the message channel to the monitoring alarm system;
the monitoring alarm system is used for acquiring a delay threshold corresponding to the service system; comparing the consumption delay information aiming at the message channel with the delay threshold value, and triggering an alarm aiming at the service system according to the comparison result.
8. An apparatus for determining consumption delay based on a distributed messaging system, comprising:
the first acquisition module is used for acquiring the message positions which can be consumed by a consumption group in a partition of a message channel and the message positions which are actually consumed by the consumption group in the partition, wherein the message positions which can be consumed by the consumption group are the latest message positions which are exposed to the consumption group by the partition, and the message positions which are actually consumed by the consumption group are synchronous with or behind the message positions which can be consumed by the consumption group;
a determining module, configured to determine first time information corresponding to a first message and second time information corresponding to a second message, where the first message is a message corresponding to a message position that can be consumed by the consumption group, the second message is a message corresponding to a message position that is actually consumed by the consumption group, and the first time information and the second time information represent the same type of time;
and the calculating module is used for calculating the difference value of the first time information and the second time information and taking the difference value as consumption delay information aiming at the subareas.
9. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any one of claims 1 to 5 when executing a program stored in the memory.
10. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of the claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010401680.3A CN111600774B (en) | 2020-05-13 | 2020-05-13 | Consumption delay determination method, system, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010401680.3A CN111600774B (en) | 2020-05-13 | 2020-05-13 | Consumption delay determination method, system, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111600774A true CN111600774A (en) | 2020-08-28 |
| CN111600774B CN111600774B (en) | 2021-11-12 |
Family
ID=72188770
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010401680.3A Active CN111600774B (en) | 2020-05-13 | 2020-05-13 | Consumption delay determination method, system, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111600774B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113792097A (en) * | 2021-01-26 | 2021-12-14 | 北京沃东天骏信息技术有限公司 | Delay trigger estimation method and device for display information, medium and electronic equipment |
| CN115567477A (en) * | 2022-09-19 | 2023-01-03 | 上海星起源新能源科技有限公司 | Method, equipment and storage medium for processing message accumulation |
| CN116132395A (en) * | 2022-11-15 | 2023-05-16 | 马上消费金融股份有限公司 | Message processing method, electronic device and computer readable storage medium |
Citations (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101207823A (en) * | 2007-11-22 | 2008-06-25 | 武汉大学 | Method for video encoding of video frequency corresponding synthesis anti-code error |
| CN101795440A (en) * | 2010-03-29 | 2010-08-04 | 中兴通讯股份有限公司 | System and method for correcting location registration information in an integrated digital enhanced network |
| CN101827440A (en) * | 2009-03-06 | 2010-09-08 | 中兴通讯股份有限公司 | Method for realizing period position updating |
| CN101839976A (en) * | 2010-04-07 | 2010-09-22 | 中国人民解放军理工大学 | Radar system error estimating method by utilizing beidou I terminal |
| CN101888689A (en) * | 2009-05-14 | 2010-11-17 | 联芯科技有限公司 | Method and device for determining PRACH (physical random access channel) sending power and method and device for responding to UpPCH (uplink pilot channel) message |
| CN102289752A (en) * | 2011-07-18 | 2011-12-21 | 北京融易通信息技术有限公司 | Mobile payment method and system based on cipher |
| CN102984673A (en) * | 2011-09-05 | 2013-03-20 | 国家宇宙研究中心 | A method for detecting Automatic Identification System (AIS) messages |
| CN103944980A (en) * | 2014-04-11 | 2014-07-23 | 东南大学 | WSID information sensing system and method used for container Internet of Things controlling |
| CN104181549A (en) * | 2014-08-18 | 2014-12-03 | 中国航空无线电电子研究所 | Method for positioning aircraft based on 1090ES data chain ground stations |
| US20150081389A1 (en) * | 2013-05-13 | 2015-03-19 | Webtrends, Inc. | Method and system that identify market segments and that facilitate targeted information distribution |
| CN106375462A (en) * | 2016-09-13 | 2017-02-01 | 北京百度网讯科技有限公司 | Method and device for realizing message persistence in distributed message system |
| CN106648904A (en) * | 2017-01-09 | 2017-05-10 | 大连理工大学 | Self-adaptive rate control method for stream data processing |
| CN108683468A (en) * | 2018-04-27 | 2018-10-19 | 河海大学常州校区 | AUV mobile data collection algorithms in underwater sensing network based on data prediction |
| CN109408501A (en) * | 2018-11-07 | 2019-03-01 | 北京锐安科技有限公司 | A kind of processing method of position data, device, server and storage medium |
| CN109493076A (en) * | 2018-11-09 | 2019-03-19 | 武汉斗鱼网络科技有限公司 | A kind of unique consuming method of Kafka message, system, server and storage medium |
| CN109739659A (en) * | 2018-12-13 | 2019-05-10 | 上海恺英网络科技有限公司 | A kind of method and system of the information consumption of Distributed Message Queue |
| US10459655B1 (en) * | 2018-04-30 | 2019-10-29 | Amazon Technologies, Inc. | Rapid volume backup generation from distributed replica |
| CN110647547A (en) * | 2019-09-20 | 2020-01-03 | 北京百分点信息科技有限公司 | Consumption delay monitoring method and device, electronic equipment and computer readable storage medium |
| US10599740B1 (en) * | 2014-09-10 | 2020-03-24 | Instart Logic, Inc. | Program code streaming |
| US10608649B1 (en) * | 2018-11-19 | 2020-03-31 | Silicon Laboratories Inc. | Relative frequency offset error and phase error detection for clocks |
| US10623789B1 (en) * | 2011-03-14 | 2020-04-14 | Vmware, Inc. | Quality evaluation of multimedia delivery in cloud environments |
-
2020
- 2020-05-13 CN CN202010401680.3A patent/CN111600774B/en active Active
Patent Citations (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101207823A (en) * | 2007-11-22 | 2008-06-25 | 武汉大学 | Method for video encoding of video frequency corresponding synthesis anti-code error |
| CN101827440A (en) * | 2009-03-06 | 2010-09-08 | 中兴通讯股份有限公司 | Method for realizing period position updating |
| CN101888689A (en) * | 2009-05-14 | 2010-11-17 | 联芯科技有限公司 | Method and device for determining PRACH (physical random access channel) sending power and method and device for responding to UpPCH (uplink pilot channel) message |
| CN101795440A (en) * | 2010-03-29 | 2010-08-04 | 中兴通讯股份有限公司 | System and method for correcting location registration information in an integrated digital enhanced network |
| CN101839976A (en) * | 2010-04-07 | 2010-09-22 | 中国人民解放军理工大学 | Radar system error estimating method by utilizing beidou I terminal |
| US10623789B1 (en) * | 2011-03-14 | 2020-04-14 | Vmware, Inc. | Quality evaluation of multimedia delivery in cloud environments |
| CN102289752A (en) * | 2011-07-18 | 2011-12-21 | 北京融易通信息技术有限公司 | Mobile payment method and system based on cipher |
| CN102984673A (en) * | 2011-09-05 | 2013-03-20 | 国家宇宙研究中心 | A method for detecting Automatic Identification System (AIS) messages |
| US20150081389A1 (en) * | 2013-05-13 | 2015-03-19 | Webtrends, Inc. | Method and system that identify market segments and that facilitate targeted information distribution |
| CN103944980A (en) * | 2014-04-11 | 2014-07-23 | 东南大学 | WSID information sensing system and method used for container Internet of Things controlling |
| CN104181549A (en) * | 2014-08-18 | 2014-12-03 | 中国航空无线电电子研究所 | Method for positioning aircraft based on 1090ES data chain ground stations |
| US10599740B1 (en) * | 2014-09-10 | 2020-03-24 | Instart Logic, Inc. | Program code streaming |
| CN106375462A (en) * | 2016-09-13 | 2017-02-01 | 北京百度网讯科技有限公司 | Method and device for realizing message persistence in distributed message system |
| CN106648904A (en) * | 2017-01-09 | 2017-05-10 | 大连理工大学 | Self-adaptive rate control method for stream data processing |
| CN108683468A (en) * | 2018-04-27 | 2018-10-19 | 河海大学常州校区 | AUV mobile data collection algorithms in underwater sensing network based on data prediction |
| US10459655B1 (en) * | 2018-04-30 | 2019-10-29 | Amazon Technologies, Inc. | Rapid volume backup generation from distributed replica |
| CN109408501A (en) * | 2018-11-07 | 2019-03-01 | 北京锐安科技有限公司 | A kind of processing method of position data, device, server and storage medium |
| CN109493076A (en) * | 2018-11-09 | 2019-03-19 | 武汉斗鱼网络科技有限公司 | A kind of unique consuming method of Kafka message, system, server and storage medium |
| US10608649B1 (en) * | 2018-11-19 | 2020-03-31 | Silicon Laboratories Inc. | Relative frequency offset error and phase error detection for clocks |
| CN109739659A (en) * | 2018-12-13 | 2019-05-10 | 上海恺英网络科技有限公司 | A kind of method and system of the information consumption of Distributed Message Queue |
| CN110647547A (en) * | 2019-09-20 | 2020-01-03 | 北京百分点信息科技有限公司 | Consumption delay monitoring method and device, electronic equipment and computer readable storage medium |
Non-Patent Citations (2)
| Title |
|---|
| CAO NGOC NGUYEN等: "《KOHA: Building a Kafka-Based Distributed Queue System on the Fly in a Hadoop Cluster》", 《: 2016 IEEE 1ST INTERNATIONAL WORKSHOPS ON FOUNDATIONS AND APPLICATIONS OF SELF SYSTEMS》 * |
| 孙大为: "《大数据流式计算关键技术及系统实例》", 《JOURNAL OF SOFTWARE 软件学报》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113792097A (en) * | 2021-01-26 | 2021-12-14 | 北京沃东天骏信息技术有限公司 | Delay trigger estimation method and device for display information, medium and electronic equipment |
| CN115567477A (en) * | 2022-09-19 | 2023-01-03 | 上海星起源新能源科技有限公司 | Method, equipment and storage medium for processing message accumulation |
| CN115567477B (en) * | 2022-09-19 | 2024-05-31 | 上海星起源新能源科技有限公司 | Method, equipment and storage medium for processing message accumulation |
| CN116132395A (en) * | 2022-11-15 | 2023-05-16 | 马上消费金融股份有限公司 | Message processing method, electronic device and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111600774B (en) | 2021-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111600774B (en) | Consumption delay determination method, system, device, equipment and readable storage medium | |
| CN110213068B (en) | Message middleware monitoring method and related equipment | |
| CN110222091B (en) | Real-time statistical analysis method for mass data | |
| CN106548402B (en) | Resource transfer monitoring method and device | |
| CN107748790B (en) | Online service system, data loading method, device and equipment | |
| US7502844B2 (en) | Abnormality indicator of a desired group of resource elements | |
| CN112311617A (en) | Configured data monitoring and alarming method and system | |
| CN112751726B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN110535713B (en) | Monitoring management system and monitoring management method | |
| CN113220534A (en) | Cluster multi-dimensional anomaly monitoring method, device, equipment and storage medium | |
| CN116804957A (en) | System monitoring method and device | |
| CN114490741A (en) | Time sorting method and device based on trusted block chain, electronic equipment and medium | |
| CN111338888B (en) | Data statistics method and device, electronic equipment and storage medium | |
| WO2017008658A1 (en) | Storage checking method and system for text data | |
| CN111538459B (en) | Method and device for determining data inclination, electronic equipment and readable storage medium | |
| CN114553747A (en) | Method, device, terminal and storage medium for detecting abnormality of redis cluster | |
| CN113835905A (en) | Message queue load balancing method and device, electronic equipment and medium | |
| CN117130851B (en) | High-performance computing cluster operation efficiency evaluation method and system | |
| CN112751722B (en) | Data transmission quality monitoring method and system | |
| CN113409876A (en) | Method and system for positioning fault hard disk | |
| CN106161058B (en) | Alarm classification method and device | |
| CN108390770B (en) | Information generation method and device and server | |
| CN113568781B (en) | Database error processing method and device and database cluster access system | |
| CN114816915A (en) | Link tracking method and device | |
| CN116074331A (en) | Block data synchronization method and related product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |