CN115378849B - Flow real-time monitoring method and device, electronic equipment and readable storage medium - Google Patents
Flow real-time monitoring method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN115378849B CN115378849B CN202211018936.8A CN202211018936A CN115378849B CN 115378849 B CN115378849 B CN 115378849B CN 202211018936 A CN202211018936 A CN 202211018936A CN 115378849 B CN115378849 B CN 115378849B
- Authority
- CN
- China
- Prior art keywords
- monitoring
- data
- redisson
- time
- flow
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/50—Testing arrangements
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/50—Reducing energy consumption in communication networks in wire-line communication networks, e.g. low power modes or reduced link rate
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application provides a method, a device, electronic equipment and a readable storage medium for monitoring traffic in real time, wherein after data traffic reported by each client is acquired, each data traffic is added to a created redisson delay queue, and the adding time when each data traffic is added is recorded. And obtaining expiration time based on the addition time and the set delay time for each data flow, moving the data flow to the created blocking queue when the expiration time of the data flow arrives, and simultaneously monitoring and acquiring the flow information of the data flow in the redisson delay queue in real time. According to the scheme, each data flow leaves the queue according to the delay time after being added to the redisson delay queue, the window corresponding to the redisson delay queue is unchanged, the window is reversely slid in real time by taking the data flow as a reference, the problem of monitoring loopholes is avoided, and the accuracy of flow monitoring can be guaranteed.
Description
Technical Field
The invention relates to the technical field of monitoring, in particular to a method and a device for monitoring flow in real time, electronic equipment and a readable storage medium.
Background
Along with the development of big data, a large amount of data traffic is often generated in each industry, and in order to manage well, the data traffic is generally monitored, so that the specific situation of various data traffic is comprehensively known.
In the prior art, a time window mode is generally adopted to monitor data traffic within a certain window width. For example, the time window to be monitored is divided into a plurality of interval barrels with equal length, and the sum of the flow counts of the interval barrels is the sum of the flow of the current total monitored interval. The time window automatically slides once after every other barrel interval time, and the current counting barrel pointer also slides forwards. When new flow comes in, the flow enters a barrel pointed by a pointer of the current counting barrel to count, and real-time statistics of the flow is ensured.
The mode adopted in the prior art can be understood as a window forward sliding mode, and the mode adopts a mode of sliding at fixed time and fixed interval, so that monitoring loopholes can be generated during each sliding, and flow monitoring errors are caused.
Disclosure of Invention
The invention aims at providing a method, a device, electronic equipment and a readable storage medium for monitoring flow in real time, which can avoid monitoring loopholes so as to ensure the accuracy of flow monitoring.
Embodiments of the invention may be implemented as follows:
in a first aspect, the present invention provides a method for monitoring traffic in real time, applied to a monitoring server, where the monitoring server communicates with a plurality of clients, the method includes:
acquiring data traffic reported by each client;
adding each data flow to the created redisson delay queue, and recording the adding time when each data flow is added;
obtaining an expiration time based on the addition time and the set delay time for each data traffic, and moving the data traffic to a created blocking queue when the expiration time of the data traffic arrives;
and monitoring and acquiring flow information of the data flow in the redisson delay queue in real time.
In an alternative embodiment, the step of adding each of the data traffic to the created redisson delay queue includes:
inquiring whether the monitoring task is configured with a state monitoring function or not;
if a state monitoring function is configured, a state corresponding to each data flow is obtained;
and adding each data traffic to the created redisson delay queue based on the corresponding state of each data traffic.
In an optional embodiment, the step of adding each data traffic to the created redisson delay queue based on the state corresponding to each data traffic includes:
inquiring whether the state monitoring function has a specified target state;
if the target state is specified, determining a target data flow with the target state based on the state corresponding to each data flow;
and adding the target data traffic to the created redisson delay queue.
In an optional embodiment, the step of monitoring and obtaining the flow information of the data flow in the redisson delay queue in real time includes:
monitoring whether the total flow of the data flows in the redisson delay queue exceeds a preset threshold value in real time;
and if the total flow amount of the data flows in the redisson delay queue exceeds the preset threshold value, carrying out alarm processing.
In an optional implementation manner, the step of performing alarm processing if the total number of data flows in the redisson delay queue exceeds the preset threshold value includes:
if the total flow of the data flow in the redisson delay queue exceeds the preset threshold, monitoring the total flow of the data flow in the redisson delay queue again after the self-checking time is set at intervals;
and if the total flow number obtained through the re-monitoring exceeds the preset threshold value, carrying out alarm processing, and if the total flow number obtained through the re-monitoring does not exceed the preset threshold value, not carrying out processing.
In an alternative embodiment, the method further comprises:
and aiming at the blocking queue, carrying out clearing treatment on the data flow in the blocking queue according to the set clearing period.
In an alternative embodiment, the method further comprises:
and sending the acquired data traffic to a third party storage system to carry out redis storage on the data traffic.
In a second aspect, the present invention provides a traffic real-time monitoring device, applied to a monitoring server, where the monitoring server communicates with a plurality of clients, the device includes:
the acquisition module is used for acquiring the data flow reported by each client;
the recording module is used for adding each data flow to the created redisson delay queue and recording the adding time when each data flow is added;
a moving module, configured to obtain, for each of the data traffic, an expiration time based on the addition time and a set delay time, and move the data traffic to a created blocking queue when the expiration time of the data traffic arrives;
and the monitoring module is used for monitoring and acquiring the flow information of the data flow in the redisson delay queue in real time.
In a third aspect, the present invention provides an electronic device comprising one or more storage media and one or more processors in communication with the storage media, the one or more storage media storing machine-executable instructions that are executable by the processor to perform the method steps recited in any one of the preceding embodiments when the electronic device is operated.
In a fourth aspect, the present invention provides a computer-readable storage medium storing machine-executable instructions which, when executed, implement the method steps of any of the preceding embodiments.
The beneficial effects of the embodiment of the invention include, for example:
the application provides a method, a device, electronic equipment and a readable storage medium for monitoring traffic in real time, wherein after data traffic reported by each client is acquired, each data traffic is added to a created redisson delay queue, and the adding time when each data traffic is added is recorded. And obtaining expiration time based on the addition time and the set delay time for each data flow, moving the data flow to the created blocking queue when the expiration time of the data flow arrives, and simultaneously monitoring and acquiring the flow information of the data flow in the redisson delay queue in real time. According to the scheme, each data flow leaves the queue according to the delay time after being added to the redisson delay queue, the window corresponding to the redisson delay queue is unchanged, the window is reversely slid in real time by taking the data flow as a reference, the problem of monitoring loopholes is avoided, and the accuracy of flow monitoring can be guaranteed.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic view of window sliding monitoring provided in the prior art;
fig. 2 is an application scenario schematic diagram of a flow real-time monitoring method provided in an embodiment of the present application;
fig. 3 is a flowchart of a flow real-time monitoring method provided in an embodiment of the present application;
FIG. 4 is a flowchart of sub-steps included in step S102 of FIG. 3;
FIG. 5 is a flow chart of sub-steps included in step S1023 of FIG. 4;
FIG. 6 is a flow chart of sub-steps included in step S104 of FIG. 3;
FIG. 7 is a flowchart of sub-steps included in step S1042 in FIG. 6;
fig. 8 is a block diagram of an electronic device according to an embodiment of the present application;
fig. 9 is a functional block diagram of a flow real-time monitoring device according to an embodiment of the present application.
Icon: 110-a storage medium; a 120-processor; 130-a flow real-time monitoring device; 131-an acquisition module; 132—a recording module; 133-a movement module; 134-a monitoring module; 140-communication interface.
Detailed Description
In the prior art, when the flow monitoring is performed, a forward sliding window mode is generally adopted. As shown in fig. 1, the time window to be monitored is split into 3 equal-length interval barrels, and the sum of the flow counts of the 3 barrels is the sum of the flow in the current total monitored interval, that is, the finer the single-barrel time division is, the more accurate the window flow data is.
The time window automatically slides once every other barrel interval time, and at the moment, the current counting barrel pointer also slides forwards. When new flow comes in, the flow enters a barrel pointed by a pointer of the current counting barrel to count, and real-time statistics of the flow is ensured. As shown in fig. 1, for example, the flow rate of approximately 3 minutes is monitored in real time, and the window of 3 minutes is split into 3 buckets with 1 minute interval. And if the barrel count number 1 is 10, the barrel count number 2 is 5 and the barrel count number 3 is 5 at the current moment, the flow rate of the near 3 minutes is 20 times. After 1 minute, the window slides, at which time barrel 1 data clears 0, barrel 4 is started, and barrel 4 counts from 0.
In the prior art, the forward sliding mode with the fixed time sliding and fixed interval is adopted for monitoring the flow. In this manner, it is assumed that sliding is performed every 1 minute for a 10 minute window. If the current window is 12 points 0 minutes to 12 points 10 minutes, sliding once after 1 minute, and changing the window into: 12 points 1 to 12 points 11 points. However, each time the slide is performed, a monitoring leak of 1 minute, namely, 12 points 0 point to 12 points 1 point is generated. If the threshold is 100, the split of 0 min-1 is 80, the split of 10 min-11 is 70, the sliding mode is not problematic, and the window of 10 minutes at 12 min 30 seconds to 12 min 30 seconds breaks through the threshold. This is also a disadvantage of forward sliding, the error depends on the interval and frequency of sliding, and maintenance of the interval and frequency is very resource-intensive.
Based on the research findings, the application provides a flow real-time monitoring scheme, and the scheme adopts a reverse sliding idea, each data flow is automatically added to the window and leaves the window after the delay time expires according to the delay time, the window is fixed, the window is reversely slid in real time by taking the data flow as a reference, the problem of monitoring loopholes does not exist, and the accuracy of flow monitoring can be ensured.
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.
Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
It should be noted that the features of the embodiments of the present invention may be combined with each other without conflict.
Referring to fig. 2, an application scenario diagram of a method for monitoring traffic in real time according to an embodiment of the present application is shown, where the application scenario includes a monitoring server, a third party storage system, and a plurality of clients. Each client can communicate with the monitoring server to realize interaction of data and information. The monitoring server may also communicate with a third party storage system to send data, etc. to the monitoring server.
In this embodiment, each client may be a terminal device, for example, a personal computer, a notebook computer, or the like. The monitoring server may be a single server or a server cluster formed by a plurality of servers. The third party storage system may be a redis storage system, which may enable persistent storage of data.
Referring to fig. 3, a flowchart of a flow real-time monitoring method according to an embodiment of the present application may be implemented by an electronic device, for example, the electronic device may be the monitoring server. The specific flow shown in fig. 3 will be described in detail.
S101, obtaining data traffic reported by each client.
S102, adding each data traffic to the created redisson delay queue, and recording the adding time when each data traffic is added.
S103, obtaining expiration time based on the addition time and the set delay time for each data flow, and moving the data flow to the created blocking queue when the expiration time of the data flow arrives.
S104, monitoring and obtaining flow information of the data flow in the redisson delay queue in real time.
In this embodiment, the data traffic generated on each client may be reported to the monitoring server. For example, taking a financial service as an example, each client may be a terminal device on the service side. Each client can record relevant data of business transaction and the like, and then report the relevant data to the monitoring server.
The client reports the data flow to the reporting channel of the monitoring server through the message middleware or through the API. In the embodiment, the multi-channel reporting of the data flow is supported, and the method has the advantage of convenience and rapidness.
In this embodiment, the configuration of the service name corresponding to the data traffic may be performed in advance, so that the service name may be carried when the data traffic is reported, so as to distinguish different services.
The monitoring server can support multi-thread and multi-instance processing, wherein the multi-instance processing can adopt a plurality of servers for parallel processing, so that the throughput of data traffic is improved.
After the monitoring server acquires the data traffic reported by the client, the data traffic is added into the established redisson delay queue. The redisson delay queue is a RDelayedQueue delay queue provided by a distributed scalable Java data structure redisson.
When each data flow is added to the redisson delay queue, the adding time of the data flow is recorded. And, a delay time is also set for the redisson delay queue, and the delay time can be understood as a monitoring window width, for example, if the data traffic within 10 minutes needs to be monitored, the delay time can be set to be 10 minutes.
And starting based on the adding time of each data flow to the redisson delay queue, and obtaining the expiration time of each data flow according to the delay time. For example, when the adding time of a certain data traffic to the redisson delay queue is 12:00 and the delay time is set to 10 minutes, the expiration time of the data traffic is 12:10. Thus, upon reaching the expiration time of the data traffic, the data traffic is moved into the created blocking queue, i.e., removed from the redisson delay queue.
In the manner described above, the data traffic stored in the redisson delay queue remains valid traffic throughout the width of the monitoring window. Therefore, the effective monitoring of the data traffic can be realized by monitoring the traffic information of the data traffic in the redisson delay queue in real time.
In the scheme, the monitoring of the data traffic flowing on the time line is realized by adopting a fixed window instead of a moving window mode, and the data traffic is moved in and out of the fixed window mode. The window is reversely slid in real time by taking the data flow as a reference object, so that the problem of monitoring loopholes in a mode of forward moving the window does not exist. A more accurate monitoring of the data traffic can be achieved.
In addition, in this embodiment, based on the RDelayedQueue delay queue provided by the distributed scalable Java data structure redisson, the window sliding time interval can be reduced to a lower level, for example, 10ms level, by implementing the real-time sliding of the time window in combination with the reverse thinking. And the concurrency security under the distributed type is supported, and the accuracy of global flow statistics can be ensured when the multi-service node reports the data flow simultaneously.
In addition, in the prior art, when data traffic is monitored, mechanical counting is often adopted, so that the data traffic cannot be classified and monitored, and all reported data traffic is identical. For example, only data traffic within nearly 10 minutes can be monitored as 100, but it is not possible to monitor what the specific successful data traffic is, what the failure is, etc.
Based on this, referring to fig. 4, in this embodiment, when the reported data traffic is acquired and added to the redisson delay queue, the following manner may be implemented:
s1021, inquiring whether the current monitoring task is configured with a state monitoring function, if so, executing the following step S1022, and if not, executing the following step S1024.
S1022, obtaining the corresponding state of each data flow.
S1023, adding each data traffic to the created redisson delay queue based on the corresponding state of each data traffic.
S1024, each data flow is directly added to the created redisson delay queue.
In this embodiment, whether a state monitoring function needs to be configured can be determined according to requirements, and the state monitoring function can monitor the state of the reported data traffic, so that the data traffic is processed based on the monitored state.
The state corresponding to each data flow may be set differently according to different service types, for example, the state of the data flow may include success, failure, processing, and the like.
If the state of the data traffic needs to be monitored, the data traffic can be added to the created redisson delay queue based on the state of each data traffic.
There may be multiple redisson delay queues, and data traffic in different states may be added to the different redisson delay queues, respectively. For example, data traffic for a successful state within approximately 10 minutes may be added to the redisson delay queue, data traffic for a failed state within approximately 10 minutes may be added to the redisson delay queue, and so on, respectively.
If the state monitoring function is not configured in advance, the data traffic can be directly added into the same redisson delay queue without distinguishing different states.
It should be noted that, in this embodiment, the state monitoring function may be preconfigured to realize different states, and the state of the data traffic may be obtained by configuring the monitoring tag. For example, when it is required to monitor both successful traffic and failed traffic of data traffic under a certain service, the monitoring tag may be configured so that the client carries the successful tag or the failed tag of each data traffic when reporting the data traffic, so that the monitoring server obtains whether the status of each data traffic is a successful status or a failed status. To monitor the successful state data traffic and the failed state data traffic under the service respectively.
On the basis of the above, in the case of monitoring data traffic, the present embodiment may need to monitor data traffic in a certain state with emphasis, but does not need to pay attention to data traffic in other states. Therefore, referring to fig. 5, the above step S1023 can be implemented by:
s10231, inquiring whether the state monitoring function has a specified target state.
S10232, if the target state is specified, determining the target data flow with the target state based on the state corresponding to each data flow.
S10233, adding the target data traffic to the created redisson delay queue.
In this embodiment, the configured state monitoring function may or may not have a specified target state. For example, the specified target state may be any one or more of a failure state and a success state, and the data traffic of one or more states is monitored. Without the specified target state, only the data traffic is state-differentiated, and the data traffic in various states is monitored.
Therefore, in the condition of not having a specified target state, data traffic in various states can be added to different redisson delay queues for monitoring respectively according to the above. And under the condition of a specified target state, the target data flow with the state of the target state in the acquired data flow needs to be determined, and the target data flow is monitored in a targeted manner.
For example, if it is desired to monitor the data traffic whose status is failed for approximately 10 minutes, the target status may be set to failed. After the monitoring server acquires the reported data traffic, determining the data traffic with the failure state, and adding the data traffic with the failure state into a redisson delay queue, so as to realize real-time monitoring of the data traffic with the failure state within about 10 minutes.
In this embodiment, in order to avoid the problem of memory resource waste, when the data traffic in the redisson delay queue reaches the expiration time, the delay consuming task may be added through a time wheel after the data traffic is moved to the created blocking queue.
And aiming at the blocking queue, clearing the data traffic in the blocking queue according to the set clearing period.
In this embodiment, the clearing period may be set according to the requirement, for example, a timely clearing mode may also be set, that is, the data traffic entering the blocking queue is cleared timely. In this embodiment, the hashaldweelter time wheel is used to realize transfer and consumption of data traffic, so that thread resources can be efficiently utilized to perform batch scheduling. The method comprises the steps of binding a large number of dispatching tasks to the same dispatcher, and using the same dispatcher to manage, trigger and run all the tasks. Various delay tasks, periodic tasks, notification tasks, and the like can be efficiently managed. And (3) processing the expiration data based on HashedWheelpimer delay consumption, so that CPU resources are saved and utilized to the greatest extent.
In addition, in this embodiment, for the data traffic within the required monitoring window width stored in the redisson delay queue, the traffic exceeding situation may occur, so, referring to fig. 6, in this embodiment, in the step of monitoring the traffic information of the data traffic in the redisson delay queue in real time, the following manner may be implemented:
s1041, monitoring whether the total flow number of the data flows in the redisson delay queue exceeds a preset threshold in real time.
S1042, if the total flow of the data flows in the redisson delay queue exceeds the preset threshold, alarm processing is performed.
In this embodiment, after a period of monitoring, the data traffic stored in the redisson delay queue is the data traffic valid during the monitoring period, for example, the data traffic within about 10 minutes. The data traffic may show problems on the service side or show the processing pressure of the data.
In this embodiment, the configuration of the maximum allowable flow value in the monitoring period may be performed in advance. If the data flow rate suddenly increases in a certain monitoring period, for example, if the data flow rate exceeds a preset configured threshold, the processing pressure may be greatly influenced, or a problem occurs in a transaction at a service side, so that in this case, an alarm processing may be performed to notify a manager to perform problem investigation.
In addition, in this embodiment, considering that a sudden increase of data traffic may occur during some busy periods of service, the sudden increase of data traffic in this case is normal, so, in order to avoid unnecessary interference caused by frequent alarm, referring to fig. 7, in the step of alarming, this embodiment may be implemented by the following steps:
s10421, if the total flow of the data flows in the redisson delay queue exceeds the preset threshold, monitoring the total flow of the data flows in the redisson delay queue again after the self-checking time is set at intervals.
And S10422, if the total flow number obtained by the re-monitoring exceeds the preset threshold value, carrying out alarm processing.
And S10423, if the total flow number obtained by the re-monitoring does not exceed the preset threshold value, not processing.
In this embodiment, the self-checking time may be preset according to the own requirement, for example, may be 1 minute, 2 minutes, or the like. If the data flow in the redisson delay queue in a certain monitoring period is monitored to exceed the standard, waiting for a period of time to monitor and judge again.
For example, if the total number of data traffic in the redisson delay queue exceeds the preset threshold at time point 12:00, the delay time is 10 minutes, that is, the data traffic within about 10 minutes is monitored, that is, the total number of data traffic representing the data traffic generated between 11:50 and 12:00 exceeds the standard.
In this case, if the set self-checking time is 1 minute, the detection may be performed after waiting for the set self-checking, that is, the detection of the total number of data traffic in the redisson delay queue is performed again at the waiting time point 12:01, so as to determine whether the total number of data traffic exceeds the preset threshold. At this point, data that entered the redisson delay queue at 11:50 to 11:51 has been removed from the redisson delay queue, as data traffic in the redisson delay queue that originally reached the expiration time has been removed from the redisson delay queue. The new data traffic is added to the redisson delay queue during the self-test time, i.e., the generated data traffic between time points 12:00 and 12:01 enters the redisson delay queue.
The number of data flows generated in the 11:50 to 11:51 time period may not be the same as the number of data flows generated in the 12:00 to 12:01 time period. Thus, the total amount of traffic in the re-monitored redisson delay queue may change.
If the total flow monitored again still exceeds the preset threshold, the data flow is indicated to be in an exceeding state continuously, and at the moment, alarm processing can be carried out. If the total number of the re-monitored traffic does not exceed the preset threshold, it indicates that the data traffic tends to be normal over time, possibly due to a normal data traffic surge caused by a traffic peak period, so that the data traffic surge may not be processed.
In this embodiment, in the step of performing the monitoring again after the interval self-checking time, the interval may be set repeatedly and the monitoring may be performed repeatedly, for example, if the total number of the data traffic still exceeds the preset threshold value during the second monitoring at the interval self-checking time, the monitoring may be performed again at the interval self-checking time to monitor whether the total number of the data traffic still exceeds the preset threshold value.
By setting the self-checking time and monitoring again at intervals of the self-checking time, the interference of frequent alarms caused by the situation of sudden increase of data flow under some normal conditions can be avoided.
In this embodiment, in order to avoid the loss of the data traffic caused by the downtime and restarting of the server, the monitoring server may send the acquired data traffic to the third party storage system to perform redis storage on the data traffic, thereby avoiding the loss of the data traffic.
In the flow real-time monitoring scheme provided by the embodiment, the window is fixed by designing a reverse window sliding mode aiming at the problem of monitoring loopholes in the forward window sliding mode adopted in the prior art, and the effective flow stored in the window in the monitoring period is ensured by entering and moving out data flow, so that the monitoring loopholes caused by window sliding can be avoided.
In addition, by configuring the state monitoring function, the state of the data traffic can be monitored, so that the data traffic of different states or the target state of interest can be monitored in a targeted manner.
Aiming at the problem that in the prior art, if a server is restarted or down, data is easy to lose in a local memory storage mode, in the scheme, data flow is stored by adopting a persistent storage component redis through a third-party storage system, so that the problem of data flow loss is avoided.
In the embodiment, based on the RDelayedQueue delay queue provided by the distributed scalable Java data structure redisson, window real-time sliding is realized by reverse thinking, and concurrency security under the distributed type is supported. And a plurality of service nodes report data at the same time, so that the accuracy of global flow statistics can be ensured.
In addition, the HashedWheelpimer time wheel is adopted to realize the movement and consumption of data traffic, and thread resources can be efficiently utilized to carry out batch scheduling. Various delay tasks, periodic tasks, notification tasks, and the like can be efficiently managed. Expiration data is processed based on HashedWheelpimer delay consumption, and unnecessary memory waste caused by message accumulation is prevented.
Referring to fig. 8, an exemplary component schematic diagram of an electronic device provided in an embodiment of the present application may be the monitoring server. The electronic device may include a storage medium 110, a processor 120, a traffic real-time monitoring apparatus 130, and a communication interface 140. In this embodiment, the storage medium 110 and the processor 120 are both located in the electronic device and are separately disposed. However, it should be understood that the storage medium 110 may also be separate from the electronic device and accessible to the processor 120 through a bus interface. Alternatively, the storage medium 110 may be integrated into the processor 120, for example, as a cache and/or general purpose registers.
The flow rate real-time monitoring device 130 may be understood as the electronic device, or the processor 120 of the electronic device, or may be understood as a software functional module that is independent of the electronic device or the processor 120 and implements the flow rate real-time monitoring method under the control of the electronic device.
As shown in fig. 9, the flow real-time monitoring device 130 may include an acquisition module 131, a recording module 132, a movement module 133, and a monitoring module 134. The functions of the respective functional modules of the flow rate real-time monitoring apparatus 130 are described in detail below.
An obtaining module 131, configured to obtain data traffic reported by each client;
it will be appreciated that the acquisition module 131 may be used to perform step S101 described above, and reference may be made to the details of the implementation of the acquisition module 131 as described above with respect to step S101.
A recording module 132, configured to add each data traffic to the created redisson delay queue, and record an addition time when each data traffic is added;
it will be appreciated that the recording module 132 may be used to perform step S102 described above, and reference may be made to the details of the implementation of the recording module 132 as described above with respect to step S102.
A moving module 133, configured to obtain, for each of the data traffic, an expiration time based on the addition time and the set delay time, and move the data traffic to a created blocking queue when the expiration time of the data traffic arrives;
it will be appreciated that the mobile module 133 may be used to perform step S103 described above, and reference may be made to the details of step S103 regarding the implementation of the mobile module 133.
The monitoring module 134 is configured to monitor and obtain, in real time, flow information of the data flow in the redisson delay queue.
It is understood that the monitoring module 134 may be used to perform the step S104 described above, and reference may be made to the details of the implementation of the monitoring module 134 regarding the step S104 described above.
In one possible implementation, the recording module 132 may be configured to:
inquiring whether the monitoring task is configured with a state monitoring function or not;
if a state monitoring function is configured, a state corresponding to each data flow is obtained;
and adding each data traffic to the created redisson delay queue based on the corresponding state of each data traffic.
In one possible implementation, the recording module 132 may be configured to:
inquiring whether the state monitoring function has a specified target state;
if the target state is specified, determining a target data flow with the target state based on the state corresponding to each data flow;
and adding the target data traffic to the created redisson delay queue.
In one possible implementation, the monitoring module 134 may be configured to:
monitoring whether the total flow of the data flows in the redisson delay queue exceeds a preset threshold value in real time;
and if the total flow amount of the data flows in the redisson delay queue exceeds the preset threshold value, carrying out alarm processing.
In one possible implementation, the monitoring module 134 may be configured to:
if the total flow of the data flow in the redisson delay queue exceeds the preset threshold, monitoring the total flow of the data flow in the redisson delay queue again after the self-checking time is set at intervals;
and if the total flow number obtained through the re-monitoring exceeds the preset threshold value, carrying out alarm processing, and if the total flow number obtained through the re-monitoring does not exceed the preset threshold value, not carrying out processing.
In one possible implementation, the flow real-time monitoring device 130 further includes a purge module that may be used to:
and aiming at the blocking queue, carrying out clearing treatment on the data flow in the blocking queue according to the set clearing period.
In one possible implementation manner, the traffic real-time monitoring device 130 further includes a sending module, where the sending module may be used to:
and sending the acquired data traffic to a third party storage system to carry out redis storage on the data traffic.
The process flow of each module in the apparatus and the interaction flow between the modules may be described with reference to the related descriptions in the above method embodiments, which are not described in detail herein.
Further, the embodiment of the application also provides a computer readable storage medium, and the computer readable storage medium stores machine executable instructions, which when executed, implement the flow real-time monitoring method provided by the embodiment.
Specifically, the computer readable storage medium can be a general-purpose storage medium, such as a mobile disk, a hard disk, or the like, and when a computer program on the computer readable storage medium is executed, the above-described flow real-time monitoring method can be executed. With respect to the processes involved in the computer readable storage medium and when executed as executable instructions thereof, reference is made to the relevant descriptions of the method embodiments described above and will not be described in detail herein.
In summary, the method, the device, the electronic device and the readable storage medium for monitoring traffic in real time provided in the embodiments of the present application add each data traffic to the created redisson delay queue after obtaining the data traffic reported by each client, and record the adding time when each data traffic is added. And obtaining expiration time based on the addition time and the set delay time for each data flow, moving the data flow to the created blocking queue when the expiration time of the data flow arrives, and simultaneously monitoring and acquiring the flow information of the data flow in the redisson delay queue in real time. According to the scheme, each data flow leaves the queue according to the delay time after being added to the redisson delay queue, the window corresponding to the redisson delay queue is unchanged, the window is reversely slid in real time by taking the data flow as a reference, the problem of monitoring loopholes is avoided, and the accuracy of flow monitoring can be guaranteed.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any changes or substitutions easily contemplated by those skilled in the art within the scope of the present invention should be included in the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. The method is characterized by being applied to a monitoring server, wherein the monitoring server is communicated with a plurality of clients, and the method comprises the following steps:
acquiring data traffic reported by each client;
adding each data flow to the created redisson delay queue, and recording the adding time when each data flow is added;
obtaining an expiration time based on the addition time and the set delay time for each data traffic, and moving the data traffic to a created blocking queue when the expiration time of the data traffic arrives;
and monitoring and acquiring flow information of the data flow in the redisson delay queue in real time.
2. The method of traffic real-time monitoring according to claim 1, wherein the step of adding each of the data traffic to the created redisson delay queue comprises:
inquiring whether the monitoring task is configured with a state monitoring function or not;
if a state monitoring function is configured, a state corresponding to each data flow is obtained;
and adding each data traffic to the created redisson delay queue based on the corresponding state of each data traffic.
3. The method for monitoring traffic in real time according to claim 2, wherein the step of adding each data traffic to the created redisson delay queue based on the state corresponding to each data traffic comprises:
inquiring whether the state monitoring function has a specified target state;
if the target state is specified, determining a target data flow with the target state based on the state corresponding to each data flow;
and adding the target data traffic to the created redisson delay queue.
4. The method for monitoring traffic in real time according to claim 1, wherein the step of obtaining traffic information of the data traffic in the redisson delay queue in real time comprises:
monitoring whether the total flow of the data flows in the redisson delay queue exceeds a preset threshold value in real time;
and if the total flow amount of the data flows in the redisson delay queue exceeds the preset threshold value, carrying out alarm processing.
5. The method for monitoring traffic in real time according to claim 4, wherein the step of performing alarm processing if the total traffic of the data traffic in the redisson delay queue exceeds the preset threshold value comprises:
if the total flow of the data flow in the redisson delay queue exceeds the preset threshold, monitoring the total flow of the data flow in the redisson delay queue again after the self-checking time is set at intervals;
and if the total flow number obtained through the re-monitoring exceeds the preset threshold value, carrying out alarm processing, and if the total flow number obtained through the re-monitoring does not exceed the preset threshold value, not carrying out processing.
6. The method for monitoring traffic in real time according to claim 1, further comprising:
and aiming at the blocking queue, carrying out clearing treatment on the data flow in the blocking queue according to the set clearing period.
7. The method for monitoring traffic in real time according to claim 1, further comprising:
and sending the acquired data traffic to a third party storage system to carry out redis storage on the data traffic.
8. A traffic real-time monitoring device, applied to a monitoring server, the monitoring server being in communication with a plurality of clients, the device comprising:
the acquisition module is used for acquiring the data flow reported by each client;
the recording module is used for adding each data flow to the created redisson delay queue and recording the adding time when each data flow is added;
a moving module, configured to obtain, for each of the data traffic, an expiration time based on the addition time and a set delay time, and move the data traffic to a created blocking queue when the expiration time of the data traffic arrives;
and the monitoring module is used for monitoring and acquiring the flow information of the data flow in the redisson delay queue in real time.
9. An electronic device comprising one or more storage media and one or more processors in communication with the storage media, the one or more storage media storing processor-executable machine-executable instructions that, when the electronic device is run, are executed by the processor to perform the method steps recited in any of claims 1-7.
10. A computer readable storage medium storing machine executable instructions which when executed implement the method steps of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211018936.8A CN115378849B (en) | 2022-08-24 | 2022-08-24 | Flow real-time monitoring method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211018936.8A CN115378849B (en) | 2022-08-24 | 2022-08-24 | Flow real-time monitoring method and device, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115378849A CN115378849A (en) | 2022-11-22 |
| CN115378849B true CN115378849B (en) | 2023-06-23 |
Family
ID=84068078
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211018936.8A Active CN115378849B (en) | 2022-08-24 | 2022-08-24 | Flow real-time monitoring method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115378849B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9066680B1 (en) * | 2009-10-15 | 2015-06-30 | Masimo Corporation | System for determining confidence in respiratory rate measurements |
| CN113301075A (en) * | 2020-05-18 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Flow control method, distributed system, device and storage medium |
| CN114157464A (en) * | 2021-11-23 | 2022-03-08 | 北京永信至诚科技股份有限公司 | Network test monitoring method and monitoring system |
| CN114827033A (en) * | 2022-04-15 | 2022-07-29 | 咪咕文化科技有限公司 | Data flow control method, device, equipment and computer readable storage medium |
| CN114884899A (en) * | 2022-07-12 | 2022-08-09 | 之江实验室 | Multi-mode core network forwarding and scheduling method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9967164B2 (en) * | 2014-09-02 | 2018-05-08 | Netscout Systems Texas, Llc | Methods and devices to efficiently determine node delay in a communication network |
-
2022
- 2022-08-24 CN CN202211018936.8A patent/CN115378849B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9066680B1 (en) * | 2009-10-15 | 2015-06-30 | Masimo Corporation | System for determining confidence in respiratory rate measurements |
| CN113301075A (en) * | 2020-05-18 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Flow control method, distributed system, device and storage medium |
| CN114157464A (en) * | 2021-11-23 | 2022-03-08 | 北京永信至诚科技股份有限公司 | Network test monitoring method and monitoring system |
| CN114827033A (en) * | 2022-04-15 | 2022-07-29 | 咪咕文化科技有限公司 | Data flow control method, device, equipment and computer readable storage medium |
| CN114884899A (en) * | 2022-07-12 | 2022-08-09 | 之江实验室 | Multi-mode core network forwarding and scheduling method and device |
Non-Patent Citations (2)
| Title |
|---|
| Haythem Yahyaoui ; Mohamed Faten Zhani."On Providing Low-cost Flow Monitoring for SDN Networks".《2020 IEEE 9th International Conference on Cloud Networking (CloudNet)》.2020,全文. * |
| 基于流计算的实时监控预警架构研究及应用;朱征;于帅鹏;赵志刚;吴金龙;;华东电力(第11期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115378849A (en) | 2022-11-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9514387B2 (en) | System and method of monitoring and measuring cluster performance hosted by an IAAS provider by means of outlier detection | |
| JP5474177B2 (en) | Distributed application monitoring | |
| US10558544B2 (en) | Multiple modeling paradigm for predictive analytics | |
| US9542291B2 (en) | Self-monitoring event-based system and method | |
| US11140029B1 (en) | Server side filtering in hybrid cloud environments | |
| US9921877B2 (en) | Intelligent auto-scaling | |
| EP2800024A1 (en) | System and methods for identifying applications in mobile networks | |
| CN109936613B (en) | Disaster recovery method and device applied to server | |
| CN109639504B (en) | Alarm information processing method and device based on cloud platform | |
| US20130219053A1 (en) | Method for improved handling of incidents in a network monitoring system | |
| US9929930B2 (en) | Reducing an amount of captured network traffic data to analyze | |
| CN109787850B (en) | Monitoring system, monitoring method and computing node | |
| CN108111499B (en) | Business processing performance optimization method and device, electronic equipment and storage medium | |
| CN110995497A (en) | Method for unified operation and maintenance in cloud computing environment, terminal device and storage medium | |
| CN111385122A (en) | Distributed system link tracking method and device, computer equipment and storage medium | |
| US20140351414A1 (en) | Systems And Methods For Providing Prediction-Based Dynamic Monitoring | |
| US20150281008A1 (en) | Automatic derivation of system performance metric thresholds | |
| CN111339466A (en) | Interface management method and device, electronic equipment and readable storage medium | |
| CN115378849B (en) | Flow real-time monitoring method and device, electronic equipment and readable storage medium | |
| EP3460769A1 (en) | System and method for managing alerts using a state machine | |
| CN114328156B (en) | Health detection method, device and equipment of protocol port and readable storage medium | |
| CN115514618A (en) | Alarm event processing method and device, electronic equipment and medium | |
| US11734086B2 (en) | Operation-based event suppression | |
| CN114816866A (en) | Fault processing method and device, electronic equipment and storage medium | |
| US10296967B1 (en) | System, method, and computer program for aggregating fallouts in an ordering system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |