CN116610537B - Data volume monitoring method, system, equipment and storage medium - Google Patents
Data volume monitoring method, system, equipment and storage medium Download PDFInfo
- Publication number
- CN116610537B CN116610537B CN202310893332.6A CN202310893332A CN116610537B CN 116610537 B CN116610537 B CN 116610537B CN 202310893332 A CN202310893332 A CN 202310893332A CN 116610537 B CN116610537 B CN 116610537B
- Authority
- CN
- China
- Prior art keywords
- statistical
- date
- threshold
- target
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 138
- 238000000034 method Methods 0.000 title claims abstract description 53
- 230000004044 response Effects 0.000 claims abstract description 14
- 238000004364 calculation method Methods 0.000 claims description 66
- 238000012216 screening Methods 0.000 claims description 14
- 230000006870 function Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 4
- 238000009825 accumulation Methods 0.000 claims description 3
- 230000008859 change Effects 0.000 abstract description 2
- 230000008569 process Effects 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 6
- 238000013523 data management Methods 0.000 description 5
- 230000003068 static effect Effects 0.000 description 5
- 238000007726 management method Methods 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- ZLIBICFPKPWGIZ-UHFFFAOYSA-N pyrimethanil Chemical compound CC1=CC(C)=NC(NC=2C=CC=CC=2)=N1 ZLIBICFPKPWGIZ-UHFFFAOYSA-N 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3058—Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
Abstract
The application discloses a data volume monitoring method, a system, equipment and a storage medium, wherein a target monitoring rule is obtained, and the target monitoring rule comprises parameters for calculating a statistic date set and a threshold interval; determining a statistical date set based on the target monitoring rule and a target date; determining a threshold interval based on the set of statistical dates and the target monitoring rule; and comparing the data quantity of the target date with the threshold interval, and generating an alarm instruction in response to the mismatching of the data quantity of the target date and the threshold interval, so as to monitor the data quantity in the table according to different dynamic monitoring parameter conditions. The method and the device can flexibly and accurately monitor the change of the table data to adapt to the monitoring requirements of different service scenes, thereby improving the monitoring accuracy.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a method, a system, an apparatus, and a storage medium for monitoring data volume.
Background
With the development of information technology and the increase of financial market traffic, company operation production business systems benefit from big data technology, and meanwhile, the accuracy and stability of big data systems are increasingly relied on.
In the current data management process, if the data quantity in the database is required to be monitored, the data quantity of the present day is generally only compared with the set upper limit and lower limit coefficients, and if the data quantity is not in the coefficient range, an alarm is given. However, in the method, the fixed upper and lower limit coefficients also cause the problems of false alarm, missing alarm and the like of the monitoring alarm in the data quantity monitoring process, and the monitoring accuracy is lacking.
Therefore, how to realize accurate monitoring of the data amount is a technical problem to be solved by those skilled in the art.
Disclosure of Invention
Based on the above problems, the application provides a data volume monitoring method, a system, a device and a storage medium, which are used for improving the accuracy of data monitoring.
In order to solve the above problems, the technical solution provided by the embodiment of the present application is as follows:
the first aspect of the present application provides a data volume monitoring method, including:
acquiring a target monitoring rule, wherein the target monitoring rule comprises parameters for calculating a statistical date set and a threshold interval;
determining a statistical date set based on the target monitoring rule and a target date;
determining a threshold interval based on the set of statistical dates and the target monitoring rule;
and comparing the data quantity of the target date with the threshold interval, and generating an alarm instruction in response to the fact that the data quantity of the target date is not matched with the threshold interval.
Optionally, the target monitoring rule includes a threshold calculating a statistical length parameter, and determining the statistical date set based on the target monitoring rule and the target date includes:
and calculating a statistical length parameter and a target date based on a threshold value, determining a statistical start date and a statistical expiration date, and determining a statistical date set according to the statistical start date and the statistical expiration date.
Optionally, the target monitoring rule includes a threshold value calculating a statistical length parameter and calculating a statistical auxiliary parameter, and the determining a statistical date set based on the target monitoring rule and a target date includes:
determining a statistical start date and a statistical expiration date based on a threshold calculation statistical length parameter, a target date and a calculation statistical auxiliary parameter, wherein the calculation statistical auxiliary parameter comprises a threshold calculation statistical offset parameter and/or a threshold calculation statistical interval parameter;
and determining a statistical date set according to the statistical starting date and the statistical expiration date.
Optionally, the target monitoring rule includes a statistical rule and a statistical parameter, and the determining a threshold interval based on the statistical date set and the target monitoring rule includes:
acquiring a data volume set corresponding to the statistical date set,
screening the data volume set based on a statistical rule and a statistical parameter to obtain a threshold statistical reference value;
and calculating a threshold interval based on the threshold statistical reference value and the threshold calculation coefficient.
Optionally, after the generating the alarm instruction, the method further includes:
responding to the alarm instruction in an unfinished state, and judging the message type of the alarm instruction;
and responding to the message type of the alarm instruction as a short message type, sending the alarm instruction to a corresponding alarm contact person, and adjusting the alarm instruction to a finished state.
Optionally, after the generating the alarm instruction, the method further includes:
responding to the alarm instruction in an unfinished state, and judging the message type of the alarm instruction;
responding to the message type of the alarm instruction as a popup window type, pushing the alarm instruction to a popup window message data table in combination with alarm contact person information, and adjusting the alarm instruction in the alarm message table to a completed state;
and responding to the alarm instruction in the popup message data table to be pushed to the client corresponding to the alarm contact information, and adjusting the alarm instruction in the popup message data table to be in a completed state.
A second aspect of the present application provides a data volume monitoring system comprising:
the target monitoring rule acquisition unit is used for acquiring a target monitoring rule, wherein the target monitoring rule comprises parameters for calculating a statistical date set and a threshold interval;
a statistic date set determining unit for determining a statistic date set based on the target monitoring rule and the target date;
a threshold interval determining unit for determining a threshold interval based on the statistical date set and the target monitoring rule;
and the alarm instruction generation unit is used for comparing the data quantity of the target date with the threshold interval and generating an alarm instruction in response to the fact that the data quantity of the target date is not matched with the threshold interval.
Optionally, the system further comprises:
the first message type judging unit is used for responding to the unfinished state of the alarm instruction and judging the message type of the alarm instruction;
the first pushing unit is used for pushing the alarm instruction to a popup message data table in combination with the alarm contact information in response to the fact that the message type of the alarm instruction is popup type, and adjusting the alarm instruction in the alarm message table to be in a completed state;
and the second pushing unit is used for responding to the alarm instruction in the popup message data table to be pushed to the client corresponding to the alarm contact information, and adjusting the alarm instruction in the popup message data table to be in a completed state.
Optionally, the system further comprises:
the second message type judging unit is used for responding to the unfinished state of the alarm instruction and judging the message type of the alarm instruction;
and the third pushing unit is used for responding to the message type of the alarm instruction as a short message type, sending the alarm instruction to a corresponding alarm contact person and adjusting the alarm instruction to a finished state.
A third aspect of the present application provides an electronic device, comprising: the data volume monitoring device comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the data volume monitoring method in any one of the first aspects when executing the computer program.
A fourth aspect of the present application provides a computer readable storage medium having instructions stored therein which, when run on a terminal device, cause the terminal device to perform the data amount monitoring method according to any one of the preceding first aspects.
Compared with the prior art, the application has the following beneficial effects:
acquiring a target monitoring rule, wherein the target monitoring rule comprises parameters for calculating a statistic date set and a threshold interval; determining a statistical date set based on the target monitoring rule and a target date; determining a threshold interval based on the set of statistical dates and the target monitoring rule; and comparing the data quantity of the target date with the threshold interval, and generating an alarm instruction in response to the mismatching of the data quantity of the target date and the threshold interval, so as to monitor the data quantity in the table according to different dynamic monitoring parameter conditions. The method and the device for monitoring the data of the target date can flexibly and accurately monitor the change of the table data so as to adapt to the monitoring requirements of different service scenes, thereby improving the monitoring accuracy.
Drawings
In order to more clearly illustrate this embodiment or the technical solutions of the prior art, the drawings that are required for the description of the embodiment or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for monitoring data volume according to an embodiment of the present application;
fig. 2 is a block diagram of a data volume monitoring system according to an embodiment of the present application.
Detailed Description
In order to make the present application better understood by those skilled in the art, the following description will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
In order to facilitate understanding of the technical solution provided by the embodiments of the present application, the following description will first explain the background technology related to the embodiments of the present application.
As described above, with the development of information technology and the increase of financial market traffic, company operation and production business systems benefit from big data technology, and at the same time, they depend on the stability of big data systems. In the current data management process, a set of script inspection data monitoring method based on a static threshold value is accumulated, namely when the data quantity in a database is monitored currently, the data quantity of today is compared with a set upper limit and lower limit coefficient, if the data quantity is not in the coefficient range, an alarm is given, but the problems of false alarm and missing alarm of the exposed monitoring alarm in the method also bring great challenges to data monitoring. In order to further improve systemization, automation and accuracy of data monitoring, the risk of data management is minimized, and dynamic threshold data monitoring is required to be realized.
The method provided by the embodiment of the application is executed by a background system, for example, the method can be executed by a data monitoring background server. The data monitoring background server can be a server device or a server cluster consisting of a plurality of servers.
In order to solve the problem, the embodiment of the application provides a data volume monitoring method, a system, equipment and a storage medium. Acquiring a target monitoring rule, wherein the target monitoring rule comprises parameters for calculating a statistic date set and a threshold interval; determining a statistical date set based on the target monitoring rule and a target date; determining a threshold interval based on the set of statistical dates and the target monitoring rule; and comparing the data quantity of the target date with the threshold interval, and generating an alarm instruction in response to the mismatching of the data quantity of the target date and the threshold interval, so as to monitor the data quantity in the table according to different dynamic monitoring parameter conditions. Developing dynamic threshold data monitoring will improve the execution frequency of data monitoring, discover data problems earlier and faster, and provide powerful guarantee for the stability of product operation and production; the data monitoring automation degree and the alarm accuracy are improved, the data management operation staff is helped to lighten the workload of manual processing of the monitoring alarm, and the processing efficiency of the monitoring alarm is improved.
In order to facilitate understanding of the data volume monitoring method provided by the embodiment of the present application, a description is given below of a scenario example of the present application.
The following describes a data amount monitoring method provided by the present application through an embodiment. Referring to fig. 1, fig. 1 is a flowchart of a data volume monitoring method provided by an embodiment of the present application, where an execution subject of the method is a server, and further, the subject may be a monitoring system in the server, and the method includes:
s101: and acquiring a target monitoring rule.
The target monitoring rule includes parameters for calculating a set of statistical dates and a threshold interval. In an actual application scene, parameters of the target monitoring rule can be flexibly configured by a user; for example, the parameters included in the target monitoring rule may be one or a combination of a plurality of statistical length N, a statistical offset L, a statistical interval M, a statistical rule, and a statistical parameter. For example, table 1 below, table 1 is a target monitoring rule detail table provided in the embodiment of the present application, the statistical length N is the interval length of the statistical sample, and the time unit schedules the working days of the calendar, such as 20 working days or 20 days; the statistical offset L is the offset length of the statistical interval, and the time unit schedules the working days of the calendar, such as 5 working days or 5 days; the statistical interval M is an interval of a statistical sample, and the time unit schedules the working days (M < N) of a calendar, such as three working days or three days, and the statistical rule comprises the accumulation of the working days or the increment of batches; the statistical parameters include maximum and minimum values, average numbers, and median numbers. It should be noted that the above parameters and specific contents of the parameters are merely examples, and the parameters may be adaptively adjusted in an actual application scenario, and the examples do not affect the protection scope.
TABLE 1 target monitoring rule details list
;
S102: a set of statistical dates is determined based on the target monitoring rules and target dates.
The target monitoring rule comprises a threshold calculation statistical length parameter, and in a possible implementation manner, the step of determining a statistical date set based on the target monitoring rule and a target date comprises the steps of A1-A2:
step A1: a statistical length parameter is calculated based on the threshold value to determine a statistical start date and a statistical expiration date.
Wherein the threshold calculation statistical length parameter is N data columns in table 1, and is used for representing the time sampling interval length used for calculating the threshold before the target date.
Corresponding to table 1, when the user selects the target monitoring rule configuration in case-1, the 20 th working day before the target date is the statistics start time, and the target date is the statistics deadline.
For example, if the user selects the target monitoring rule configuration in case-1 for 8 months and 3 days today, the user takes 7 months and 13 days as the statistical starting time, and 8 months and 3 days as the statistical deadline.
Step A2: and determining a statistical date set according to the statistical starting date and the statistical expiration date.
And combining the statistical start date and the statistical expiration date to obtain a statistical date set. That is, the range determination formula of the statistical date set can be D- (N+L). Ltoreq.date.ltoreq.D-L, wherein D is the target date and date is the statistical date set.
In one possible implementation manner, the determining a set of statistical dates based on the target monitoring rule and the target date includes B1-B2:
step B1: a statistical length parameter and a statistical auxiliary parameter are calculated based on the threshold value, and a statistical start date and a statistical expiration date are determined.
The calculating the statistical auxiliary parameter comprises calculating a statistical offset parameter by a threshold value and/or calculating a statistical interval parameter by the threshold value.
Corresponding to table 1, when the user selects the target monitoring rule configuration in case-3, the (20+5) th working day before the target date is the statistics start time, and the target date is the statistics deadline.
Step B2: and determining a statistical date set according to the statistical starting date and the statistical expiration date.
In a possible implementation manner, corresponding to table 1, when the user selects the target monitoring rule configuration in case-4, the 20 th working day before the target date is the statistical start time, and the target date is the statistical deadline. And then, taking 1 time sample from the time interval formed by the determined statistical starting time and the statistical ending time every 3 working days, and combining the acquired time samples to form a statistical date set.
Corresponding to table 1, when the user selects the target monitoring rule configuration in case-5, the (20+5) th day before the target date is the statistical start time, and the target date is the statistical deadline. And then, taking 1 time sample every 3 days from the time interval formed by the determined statistical starting time and the statistical ending time, and combining the acquired time samples to form a statistical date set.
S103: a threshold interval is determined based on the set of statistical dates and the target monitoring rule.
The target monitoring rule comprises a threshold value calculation statistical length parameter and a calculation statistical auxiliary parameter
In one possible implementation, the determining a threshold interval based on the set of statistical dates and the target monitoring rule includes C1-C3:
step C1: and acquiring a data volume set corresponding to the statistical date set.
And acquiring a data volume set corresponding to each statistic date determined in the step S102. For example, if the set of statistical dates has been determined to be 7 months 13 days to 8 months 3 days, data between 7 months 13 days and 8 months 3 days is acquired.
Step C2: and screening the data quantity set based on the statistical rule and the statistical parameter to obtain a threshold statistical reference value.
The statistical rule corresponds to the data in the last-last column in the table 1, the statistical parameter corresponds to the data in the last column in the table 1, the statistical rule and the statistical parameter are both used for screening conditions for the data quantity set, the statistical rule corresponds to screening of the data updating mode, and the statistical parameter corresponds to screening of the data type.
Step C3: and calculating a threshold interval based on the threshold statistical reference value and the threshold calculation coefficient.
The following are illustrated corresponding to table 1 above:
if the configuration of the target monitoring rule corresponding to CASE-1 is selected: if the latest batch at the statistical time is 3, the data updating amount of the third batch of each working day between the target date and the first 20 working days is taken as a sample, the maximum value and the minimum value are taken as threshold statistical reference values, and the data updating amount is compared with the data amount of the day table.
If the configuration of the target monitoring rule corresponding to CASE-2 is selected: if the latest batch at the counting time is 3, the accumulated data updating quantity of the third batch is taken as a sample after each working day from the target date to the first 20 working days, an average value is taken as a threshold counting reference value, and the accumulated data updating quantity is compared with the data quantity of the day table.
If the configuration of the target monitoring rule corresponding to CASE-3 is selected: if the latest batch at the counting time is 3, counting the data updating quantity of the third batch of each working day from the first 5 working days to the first (20+5) working days, taking the data updating quantity as a sample, and comparing the median with the data quantity of the day table.
If the configuration of the target monitoring rule corresponding to CASE-4 is selected: if the latest batch at the counting time is 3, 1 time sample is taken every 3 working days from the target date to the first 20 working days, then the data updating quantity of the third batch of each time sample is taken as a sample, the median is taken as a threshold counting reference value, and the data quantity of the day table is compared.
If the configuration of the target monitoring rule corresponding to CASE-5 is selected: counting the time samples from the first 5 days to the first (20+5) days, taking 1 time sample every 3 days, taking the data updating quantity of the third batch of each time sample as a sample, taking the median as a threshold value statistical reference value, and comparing with the data quantity of the day table.
In an actual application scene, if the day is 8 months and 3 days, a calendar is regularly scheduled to select natural days, when a target monitoring rule selected by a user is configured in case-1, the rear end sql is grouped by a date field, data between 7 months and 13 days and 8 months and 3 days are taken, the data of the third batch of data update quantity count (1) is obtained every day, the total value of 20 counts (1) is obtained, and then the maximum value and the minimum value in the 20 counts (1) are selected as reference values of threshold statistics according to the maximum and minimum values of statistical parameters. And respectively multiplying the threshold coefficient by the threshold coefficient, comparing the threshold coefficient with the value of the count (1) on the day of 8 months and 3 days, and if the value of the count (1) on the day of 8 months and 3 days exceeds the threshold range, producing an alarm record on the monitoring alarm page, wherein the user can process the alarm.
In one possible implementation manner, the core algorithm of the dynamic threshold data monitoring is to compare the magnitude relation between the target time data quantity and the dynamic threshold to perform monitoring alarm, and the specific model can be expressed as follows:
（1）
wherein, F in the formula (1) represents a dynamic threshold statistical calculation function, which can be four statistical calculation functions of maximum calculation, minimum calculation, median calculation and average calculation; SL represents a threshold lower limit calculation coefficient, SU represents a threshold upper limit calculation coefficient; d represents the monitoring execution target date, N represents the threshold calculation statistical length, L represents the threshold calculation statistical offset, and M represents the threshold calculation statistical interval; count represents the target date data amount calculation function.
It should be noted that the above parameters and specific contents of the parameters are merely examples, and the parameters may be adaptively adjusted in an actual application scenario, and the examples do not affect the protection scope. In one possible implementation manner, the rule management module in the monitoring system can be used for performing addition, deletion, modification and check on the data monitoring rule, mainly defining and querying, and realizing dynamic management on parameters including, but not limited to, a dynamic threshold value statistic calculation function, a threshold value lower limit calculation coefficient, a threshold value upper limit calculation coefficient, a threshold value calculation statistic length, a threshold value calculation statistic offset, a threshold value calculation statistic interval, a rule execution target date, a target date data amount calculation function, a target data source, a target data table and the like.
S104: and comparing the data quantity of the target date with the threshold interval, and generating an alarm instruction in response to the fact that the data quantity of the target date is not matched with the threshold interval.
In the practical application scenario, the acquisition of the data amount of the target date can be realized by an external data acquisition module in the system, and the external data acquisition module comprises three sub-modules of kafka (high throughput open source stream processing platform) message data acquisition, api interface data (Application Programming Interface ) acquisition and database data acquisition. Wherein, kafka is an open source stream processing platform developed by Apache software Foundation and is written by Scala and Java. In this case, the Database data acquisition service application manages ten or more different data sources, including Mysql (relational Database management system) Database, oracle (relational Database management system with distributed Database as a core provided by Oracle Database, oracle corporation) Database, and dream Database, and the data volume of the acquisition target data table is called by the rule service module, so as to calculate and generate a dynamic threshold value, and compare with the current time data volume. The module decouples the external data acquisition from the processes of rule inquiry, rule execution and the like, avoids logic confusion and increases code reusability.
In one possible implementation manner, after the generating the alarm instruction, the method further includes steps D10-D20:
step D10: and judging the message type of the alarm instruction in response to the alarm instruction being in an unfinished state.
In an actual application scenario, the alarm instruction generated in the above step may be written into an alarm message data table, and an alarm service module responsible for managing the alarm message data table polls the alarm message data table at regular time, acquires an alarm message with an incomplete state, and determines a message type.
Step D20: and responding to the message type of the alarm instruction as a short message type, sending the alarm instruction to a corresponding alarm contact person, and adjusting the alarm instruction to a finished state.
If the message is a message of a short message type, calling a short message sending service, sending the short message to a corresponding alarm contact person, and changing the state of the message in an alarm message table into completion after the short message is sent successfully.
In one possible implementation manner, after the generating the alarm instruction, the method further includes step D11-step D31:
step D11: responding to the alarm instruction in an unfinished state, and judging the message type of the alarm instruction;
step D21: and in response to the message type of the alarm instruction being a popup type, pushing the alarm instruction to a popup message data table in combination with alarm contact information, and adjusting the alarm instruction in the alarm message table to a completed state.
If the message is the popup type message, generating popup message by combining with the alarm contact person, pushing the popup message to a popup message data table, and changing the message state in the alarm message table to be completed after pushing is completed.
Step D31: and responding to the alarm instruction in the popup message data table to be pushed to the client corresponding to the alarm contact information, and adjusting the alarm instruction in the popup message data table to be in a completed state.
Meanwhile, the user terminal polls the background popup message interface at regular time, the interface scans the popup message data table, acquires the popup message with unfinished state and returns the popup message to the user terminal, and after the popup message is successfully returned, the state of the message in the popup message table is changed to be finished.
In conclusion, developing dynamic threshold data monitoring improves the execution frequency of data monitoring, discovers data problems earlier and faster, and provides powerful guarantee for the operation and production stability of products; the data monitoring automation degree and the alarm accuracy are improved, the data management operation staff is helped to lighten the workload of manual processing of the monitoring alarm, and the processing efficiency of the monitoring alarm is improved. Developing dynamic threshold data monitoring requires corresponding technical methods to support threshold calculation and monitoring alarms. The application discloses a data monitoring technical method based on a dynamic threshold value, which has the characteristics of multisource multiscale, dynamic and static combination, flexible configuration and accurate alarm.
Compared with static threshold data monitoring, the dynamic threshold is added, and an efficient technical solution is formed for data threshold monitoring; based on the decoupling design of rule service, data service and alarm service, the method has clear hierarchy, simultaneously supports the combination of static threshold and dynamic threshold, has high efficiency of process treatment, high availability of modules and more flexible expansion, and has the characteristics of multisource multi-table, dynamic and static combination, flexible configuration and accurate alarm.
The above is some specific implementation manners of the data volume monitoring method provided by the embodiment of the present application, and based on this, the present application further provides a corresponding system for data volume monitoring. The system provided by the embodiment of the application will be described from the aspect of functional modularization. Fig. 2 is a block diagram of a data volume monitoring system according to an embodiment of the present application.
The system comprises:
a target monitoring rule obtaining unit 201, configured to obtain a target monitoring rule, where the target monitoring rule includes parameters for calculating a statistical date set and a threshold interval;
a statistic date set determining unit 202 for determining a statistic date set based on the target monitoring rule and target date;
a threshold interval determining unit 203, configured to determine a threshold interval based on the statistical date set and the target monitoring rule;
and the alarm instruction generating unit 204 is configured to compare the data amount of the target date with the threshold interval, and generate an alarm instruction in response to the data amount of the target date not matching with the threshold interval.
Optionally, the system further comprises:
the first message type judging unit is used for responding to the unfinished state of the alarm instruction and judging the message type of the alarm instruction;
the first pushing unit is used for pushing the alarm instruction to a popup message data table in combination with the alarm contact information in response to the fact that the message type of the alarm instruction is popup type, and adjusting the alarm instruction in the alarm message table to be in a completed state;
and the second pushing unit is used for responding to the alarm instruction in the popup message data table to be pushed to the client corresponding to the alarm contact information, and adjusting the alarm instruction in the popup message data table to be in a completed state.
Optionally, the system further comprises:
the second message type judging unit is used for responding to the unfinished state of the alarm instruction and judging the message type of the alarm instruction;
and the third pushing unit is used for responding to the message type of the alarm instruction as a short message type, sending the alarm instruction to a corresponding alarm contact person and adjusting the alarm instruction to a finished state.
The embodiment of the application also provides corresponding equipment and a computer storage medium, which are used for realizing the data volume monitoring method scheme provided by the embodiment of the application.
Optionally, the statistical date set determining unit is specifically configured to: and calculating a statistical length parameter and a target date based on a threshold value, determining a statistical start date and a statistical expiration date, and determining a statistical date set according to the statistical start date and the statistical expiration date.
Optionally, the statistical date set determining unit is specifically configured to: determining a statistical start date and a statistical expiration date based on a threshold calculation statistical length parameter, a target date and a calculation statistical auxiliary parameter, wherein the calculation statistical auxiliary parameter comprises a threshold calculation statistical offset parameter and/or a threshold calculation statistical interval parameter; and determining a statistical date set according to the statistical starting date and the statistical expiration date.
Optionally, the threshold interval determining unit is specifically configured to: acquiring a data volume set corresponding to the statistical date set, and screening the data volume set based on a statistical rule and a statistical parameter to obtain a threshold statistical reference value; and calculating a threshold interval based on the threshold statistical reference value and the threshold calculation coefficient.
The device comprises a memory and a processor, wherein the memory is used for storing instructions or codes, and the processor is used for executing the instructions or codes so as to enable the device to execute the data volume monitoring method according to any embodiment of the application.
The computer storage medium stores code, and when the code is executed, a device executing the code implements the data volume monitoring method according to any embodiment of the present application.
It should be noted that, in the present description, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different manner from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the system or device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant points refer to the description of the method section.
It should be understood that in the present application, "at least one (item)" means one or more, and "a plurality" means two or more. "and/or" for describing the association relationship of the association object, the representation may have three relationships, for example, "a and/or B" may represent: only a, only B and both a and B are present, wherein a, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b or c may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.
It is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software modules may be disposed in Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (7)
1. A method of monitoring data volume, comprising:
acquiring a target monitoring rule, wherein the target monitoring rule comprises parameters for calculating a statistical date set and a threshold interval;
determining a statistical date set based on the target monitoring rule and a target date;
determining a threshold interval based on the set of statistical dates and the target monitoring rule;
comparing the data volume of the target date with the threshold interval, and generating an alarm instruction in response to the mismatching of the data volume of the target date with the threshold interval;
the target monitoring rule includes a threshold calculation statistical length parameter, and the determining a statistical date set based on the target monitoring rule and a target date includes:
calculating a statistical length parameter and a target date based on a threshold value, determining a statistical start date and a statistical expiration date, and determining a statistical date set according to the statistical start date and the statistical expiration date;
the target monitoring rule comprises a threshold value calculation statistical length parameter and a calculation statistical auxiliary parameter, and the method for determining a statistical date set based on the target monitoring rule and a target date comprises the following steps:
determining a statistical start date and a statistical expiration date based on a threshold calculation statistical length parameter, a target date and a calculation statistical auxiliary parameter, wherein the calculation statistical auxiliary parameter comprises a threshold calculation statistical offset parameter and/or a threshold calculation statistical interval parameter;
determining a statistical date set according to the statistical start date and the statistical expiration date;
the target monitoring rule includes a statistical rule and a statistical parameter, and the determining a threshold interval based on the statistical date set and the target monitoring rule includes:
acquiring a data volume set corresponding to the statistical date set,
screening the data volume set based on a statistical rule and a statistical parameter to obtain a threshold statistical reference value, wherein the statistical rule is used for screening a data update mode of the data volume, including accumulation on the same day or batch increment, and the statistical parameter is used for screening a data type of the data volume, including any one of a maximum value, a minimum value, an average value and a median;
calculating a threshold interval based on the threshold statistical reference value and the threshold calculation coefficient;
the formula for calculating the threshold interval is specifically as follows:
;
wherein F is a calculation function for calculating any one of a maximum value, a minimum value, a median and an average number; SL is the calculation coefficient used for adjusting the lower threshold limit, and SU is the calculation coefficient used for adjusting the upper threshold limit; d is a target date, N is a threshold value calculation statistical length parameter, L is a threshold value calculation statistical offset parameter, M is a threshold value calculation statistical interval parameter, count represents a target date data amount calculation function, count (i) is a data amount obtained by screening a data amount data update mode based on a statistical rule, and Count (t) is a data amount of a target date.
2. The method of claim 1, wherein after generating the alert instruction, further comprising:
responding to the alarm instruction in an unfinished state, and judging the message type of the alarm instruction;
and responding to the message type of the alarm instruction as a short message type, sending the alarm instruction to a corresponding alarm contact person, and adjusting the alarm instruction to a finished state.
3. The method of claim 1, wherein after generating the alert instruction, further comprising:
responding to the alarm instruction in an unfinished state, and judging the message type of the alarm instruction;
responding to the message type of the alarm instruction as a popup window type, pushing the alarm instruction to a popup window message data table in combination with alarm contact person information, and adjusting the alarm instruction in the alarm message table to a completed state;
and responding to the alarm instruction in the popup message data table to be pushed to the client corresponding to the alarm contact information, and adjusting the alarm instruction in the popup message data table to be in a completed state.
4. A data volume monitoring system, the system comprising:
the target monitoring rule acquisition unit is used for acquiring a target monitoring rule, wherein the target monitoring rule comprises parameters for calculating a statistical date set and a threshold interval;
a statistic date set determining unit for determining a statistic date set based on the target monitoring rule and the target date;
a threshold interval determining unit for determining a threshold interval based on the statistical date set and the target monitoring rule;
the alarm instruction generation unit is used for comparing the data quantity of the target date with the threshold interval and generating an alarm instruction in response to the fact that the data quantity of the target date is not matched with the threshold interval;
the statistical date set determining unit is specifically configured to: calculating a statistical length parameter and a target date based on a threshold value, determining a statistical start date and a statistical expiration date, and determining a statistical date set according to the statistical start date and the statistical expiration date;
the statistical date set determining unit is specifically configured to: determining a statistical start date and a statistical expiration date based on a threshold calculation statistical length parameter, a target date and a calculation statistical auxiliary parameter, wherein the calculation statistical auxiliary parameter comprises a threshold calculation statistical offset parameter and/or a threshold calculation statistical interval parameter; determining a statistical date set according to the statistical start date and the statistical expiration date;
the threshold interval determining unit is specifically configured to: acquiring a data volume set corresponding to the statistical date set, and screening the data volume set based on a statistical rule and a statistical parameter to obtain a threshold statistical reference value; calculating a threshold interval based on the threshold statistical reference value and the threshold calculation coefficient; the statistical rule is used for screening a data update mode of the data quantity, including accumulation of the day or batch increment, and the statistical parameter is used for screening a data type of the data quantity, including any one of maximum value, minimum value, average number and median;
the formula for calculating the threshold interval is specifically as follows:
;
wherein F is a calculation function for calculating any one of a maximum value, a minimum value, a median and an average number; SL is the calculation coefficient used for adjusting the lower threshold limit, and SU is the calculation coefficient used for adjusting the upper threshold limit; d is a target date, N is a threshold value calculation statistical length parameter, L is a threshold value calculation statistical offset parameter, M is a threshold value calculation statistical interval parameter, count represents a target date data amount calculation function, count (i) is a data amount obtained by screening a data amount data update mode based on a statistical rule, and Count (t) is a data amount of a target date.
5. The system of claim 4, wherein the system further comprises:
the first message type judging unit is used for responding to the unfinished state of the alarm instruction and judging the message type of the alarm instruction;
the first pushing unit is used for pushing the alarm instruction to a popup message data table in combination with the alarm contact information in response to the fact that the message type of the alarm instruction is popup type, and adjusting the alarm instruction in the alarm message table to be in a completed state;
and the second pushing unit is used for responding to the alarm instruction in the popup message data table to be pushed to the client corresponding to the alarm contact information, and adjusting the alarm instruction in the popup message data table to be in a completed state.
6. An electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the data volume monitoring method according to any one of claims 1-3 when the computer program is executed.
7. A computer readable storage medium, characterized in that the computer readable storage medium has stored therein instructions, which when run on a terminal device, cause the terminal device to perform the data volume monitoring method according to any of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310893332.6A CN116610537B (en) | 2023-07-20 | 2023-07-20 | Data volume monitoring method, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310893332.6A CN116610537B (en) | 2023-07-20 | 2023-07-20 | Data volume monitoring method, system, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116610537A CN116610537A (en) | 2023-08-18 |
| CN116610537B true CN116610537B (en) | 2023-11-17 |
Family
ID=87684024
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310893332.6A Active CN116610537B (en) | 2023-07-20 | 2023-07-20 | Data volume monitoring method, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116610537B (en) |
Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2003005279A1 (en) * | 2001-07-03 | 2003-01-16 | Altaworks Corporation | System and methods for monitoring performance metrics |
| WO2005045739A2 (en) * | 2003-10-24 | 2005-05-19 | Microsoft Corporation | Scalable synchronous and asynchronous processing of monitoring rules |
| CA2775419A1 (en) * | 2011-07-01 | 2013-01-01 | The Boeing Company | Method, monitoring system and computer program product for monitoring the health of a monitored system utilizing an associative memory |
| WO2016016926A1 (en) * | 2014-07-28 | 2016-02-04 | 株式会社日立製作所 | Management calculator and method for evaluating performance threshold value |
| CN105406991A (en) * | 2015-10-26 | 2016-03-16 | 上海华讯网络系统有限公司 | Method and system for generating service threshold by historical data based on network monitoring indexes |
| CN106557401A (en) * | 2016-10-13 | 2017-04-05 | 中国铁道科学研究院电子计算技术研究所 | A kind of dynamic threshold establishing method and system of information technoloy equipment monitor control index |
| CN108491310A (en) * | 2018-03-26 | 2018-09-04 | 北京九章云极科技有限公司 | A kind of daily record monitoring method and system |
| CN111190794A (en) * | 2019-12-30 | 2020-05-22 | 天津浪淘科技股份有限公司 | Operation and maintenance monitoring and management system |
| CN111199018A (en) * | 2019-12-27 | 2020-05-26 | 东软集团股份有限公司 | Abnormal data detection method and device, storage medium and electronic equipment |
| CN112445685A (en) * | 2020-11-27 | 2021-03-05 | 平安普惠企业管理有限公司 | Method, device and storage medium for dynamically updating alarm threshold |
| CN112631761A (en) * | 2020-12-31 | 2021-04-09 | 中国农业银行股份有限公司 | Task scheduling monitoring method and device |
| WO2021105927A1 (en) * | 2019-11-28 | 2021-06-03 | Mona Labs Inc. | Machine learning performance monitoring and analytics |
| US11132373B1 (en) * | 2019-04-30 | 2021-09-28 | Splunk Inc. | Decoupled update cycle and disparate search frequency dispatch for dynamic elements of an asset monitoring and reporting system |
| CN113448798A (en) * | 2020-12-30 | 2021-09-28 | 北京新氧科技有限公司 | Log data monitoring method and related equipment |
| CN113448805A (en) * | 2021-06-29 | 2021-09-28 | 中国工商银行股份有限公司 | Monitoring method, device and equipment based on CPU dynamic threshold and storage medium |
| WO2023115931A1 (en) * | 2021-12-21 | 2023-06-29 | 浪潮通信信息系统有限公司 | Big-data component parameter adjustment method and apparatus, and electronic device and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6999895B2 (en) * | 2003-03-21 | 2006-02-14 | International Business Machines Corporation | Method and structure for dynamic sampling method in on-line process monitoring |
| US9705754B2 (en) * | 2012-12-13 | 2017-07-11 | Level 3 Communications, Llc | Devices and methods supporting content delivery with rendezvous services |
| US10013329B2 (en) * | 2016-04-28 | 2018-07-03 | International Business Machines Corporation | Dynamic tracing using ranking and rating |
| US11269717B2 (en) * | 2019-09-24 | 2022-03-08 | Sap Se | Issue-resolution automation |
-
2023
- 2023-07-20 CN CN202310893332.6A patent/CN116610537B/en active Active
Patent Citations (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2003005279A1 (en) * | 2001-07-03 | 2003-01-16 | Altaworks Corporation | System and methods for monitoring performance metrics |
| WO2005045739A2 (en) * | 2003-10-24 | 2005-05-19 | Microsoft Corporation | Scalable synchronous and asynchronous processing of monitoring rules |
| CA2775419A1 (en) * | 2011-07-01 | 2013-01-01 | The Boeing Company | Method, monitoring system and computer program product for monitoring the health of a monitored system utilizing an associative memory |
| WO2016016926A1 (en) * | 2014-07-28 | 2016-02-04 | 株式会社日立製作所 | Management calculator and method for evaluating performance threshold value |
| CN105406991A (en) * | 2015-10-26 | 2016-03-16 | 上海华讯网络系统有限公司 | Method and system for generating service threshold by historical data based on network monitoring indexes |
| CN106557401A (en) * | 2016-10-13 | 2017-04-05 | 中国铁道科学研究院电子计算技术研究所 | A kind of dynamic threshold establishing method and system of information technoloy equipment monitor control index |
| CN108491310A (en) * | 2018-03-26 | 2018-09-04 | 北京九章云极科技有限公司 | A kind of daily record monitoring method and system |
| US11132373B1 (en) * | 2019-04-30 | 2021-09-28 | Splunk Inc. | Decoupled update cycle and disparate search frequency dispatch for dynamic elements of an asset monitoring and reporting system |
| WO2021105927A1 (en) * | 2019-11-28 | 2021-06-03 | Mona Labs Inc. | Machine learning performance monitoring and analytics |
| CN111199018A (en) * | 2019-12-27 | 2020-05-26 | 东软集团股份有限公司 | Abnormal data detection method and device, storage medium and electronic equipment |
| CN111190794A (en) * | 2019-12-30 | 2020-05-22 | 天津浪淘科技股份有限公司 | Operation and maintenance monitoring and management system |
| CN112445685A (en) * | 2020-11-27 | 2021-03-05 | 平安普惠企业管理有限公司 | Method, device and storage medium for dynamically updating alarm threshold |
| CN113448798A (en) * | 2020-12-30 | 2021-09-28 | 北京新氧科技有限公司 | Log data monitoring method and related equipment |
| CN112631761A (en) * | 2020-12-31 | 2021-04-09 | 中国农业银行股份有限公司 | Task scheduling monitoring method and device |
| CN113448805A (en) * | 2021-06-29 | 2021-09-28 | 中国工商银行股份有限公司 | Monitoring method, device and equipment based on CPU dynamic threshold and storage medium |
| WO2023115931A1 (en) * | 2021-12-21 | 2023-06-29 | 浪潮通信信息系统有限公司 | Big-data component parameter adjustment method and apparatus, and electronic device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116610537A (en) | 2023-08-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021164465A1 (en) | Intelligent early warning method and system | |
| CN107358247B (en) | Method and device for determining lost user | |
| CN109961198B (en) | Associated information generation method and device | |
| US20190114711A1 (en) | Financial analysis system and method for unstructured text data | |
| WO2014036442A1 (en) | System and method for predicting customer attrition using dynamic user interaction data | |
| CN111522968B (en) | Knowledge graph fusion method and device | |
| CN116610537B (en) | Data volume monitoring method, system, equipment and storage medium | |
| US11636377B1 (en) | Artificial intelligence system incorporating automatic model updates based on change point detection using time series decomposing and clustering | |
| US11651271B1 (en) | Artificial intelligence system incorporating automatic model updates based on change point detection using likelihood ratios | |
| CN113220705A (en) | Slow query identification method and device | |
| CN110855484A (en) | Method, system, electronic device and storage medium for automatically detecting traffic change | |
| CN115630861A (en) | Quality inspection strategy self-adaptive adjusting method, device and storage medium | |
| CN114493054A (en) | Carbon trading market intelligent analysis method and system based on big data | |
| CN114969187A (en) | Data analysis system and method | |
| CN113761082A (en) | Data visualization method, device and system | |
| CN114168595A (en) | Data analysis method and device | |
| CN113761390A (en) | Method and system for analyzing attribute intimacy | |
| CN113672660A (en) | Data query method, device and equipment | |
| CN110555537A (en) | Multi-factor multi-time point correlated prediction | |
| CN111737281B (en) | Database query method, device, electronic equipment and readable storage medium | |
| CN117252345B (en) | Charging method and system based on charging logic sequence | |
| CN113362097B (en) | User determination method and device | |
| CN116503175A (en) | Product risk early warning method and device | |
| CN115757074A (en) | Method for counting user operation behavior data in real time | |
| CN117633018A (en) | Query method, query device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |