CN114595118A - Data monitoring method and device, storage medium and electronic equipment - Google Patents

Data monitoring method and device, storage medium and electronic equipment Download PDF

Info

Publication number
CN114595118A
CN114595118A CN202210219898.6A CN202210219898A CN114595118A CN 114595118 A CN114595118 A CN 114595118A CN 202210219898 A CN202210219898 A CN 202210219898A CN 114595118 A CN114595118 A CN 114595118A
Authority
CN
China
Prior art keywords
monitoring
feature
data
features
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210219898.6A
Other languages
Chinese (zh)
Inventor
罗昕
陆俊峰
陈壘
魏慷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Netease Cloud Music Technology Co Ltd
Original Assignee
Hangzhou Netease Cloud Music Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Netease Cloud Music Technology Co Ltd filed Critical Hangzhou Netease Cloud Music Technology Co Ltd
Priority to CN202210219898.6A priority Critical patent/CN114595118A/en
Publication of CN114595118A publication Critical patent/CN114595118A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
    • G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
    • G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
    • G06F11/3082—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting the data filtering being achieved by aggregating or compressing the monitored data

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The disclosure relates to the technical field of internet, and particularly discloses a data monitoring method and device, a storage medium and an electronic device. The method comprises the following steps: acquiring data information of a data source to be monitored; performing feature extraction on the data information to generate monitoring features and monitoring; counting the number of the monitoring features, and determining target monitoring features according to the counting result; and analyzing time trend for each target monitoring feature to obtain the feature development trend of each target monitoring feature, and monitoring and early warning the target monitoring features according to the feature development trend. The method and the device perform monitoring on the monitoring features extracted from the data information, realize the monitoring capability of the data information of different data sources, and have comprehensive data monitoring and high flexibility.

Description

Data monitoring method and device, storage medium and electronic equipment
Technical Field
Embodiments of the present disclosure relate to the field of internet technologies, and in particular, to a data monitoring method, a data monitoring apparatus, a computer-readable storage medium, and an electronic device.
Background
This section is intended to provide a background or context to the embodiments of the disclosure and the description herein is not an admission that it is prior art, nor is it admitted to be prior art by inclusion in this section.
With the development of internet technology, network traffic is increasing continuously, various applications are in endless, and users may have abnormal events in the application using process, such as incapability of logging in, blocking, abnormal application version update, and the like, which may result in abnormal operation of the application or loss of some functions.
In the related art, data information generated during the operation of an application is monitored to find problems and perform early warning and processing.
Disclosure of Invention
In this context, embodiments of the present disclosure are intended to provide a data monitoring method, a data monitoring apparatus, a computer-readable storage medium, and an electronic device.
According to a first aspect of embodiments of the present disclosure, there is provided a data monitoring method, including: acquiring data information of a data source to be monitored; performing feature extraction on the data information to generate monitoring features and monitoring; counting the number of the monitoring features, and determining target monitoring features according to the counting result; and analyzing time trend for each target monitoring feature to obtain the feature development trend of each target monitoring feature, and monitoring and early warning the target monitoring features according to the feature development trend.
In an optional embodiment, the monitoring and early warning the target monitoring feature according to the feature development trend includes: and performing feature aggregation on a plurality of target monitoring features of which the feature development trends meet preset aggregation conditions, and taking feature aggregation results as early warning information.
In an optional implementation manner, the performing feature extraction on the data information, generating a monitoring feature, and monitoring includes: extracting the monitoring features from the data information corresponding to each user according to a preset feature extraction rule; aggregating the monitoring characteristics of all users to obtain an aggregated characteristic set; performing a monitoring operation on the monitoring features in the aggregated feature set.
In an optional implementation manner, the extracting, according to the preset feature extraction rule, the monitoring feature from the data information corresponding to each user includes: determining the running features to be searched based on the preset feature extraction rule, wherein the preset feature extraction rule is a rule for searching the running features of the application program from the data information; according to the operating characteristics to be searched, carrying out data characteristic analysis on the data information to obtain characteristic result information of the operating characteristics to be searched; and taking the operating characteristics to be searched and the corresponding characteristic result information as the monitoring characteristics.
In an optional implementation manner, the extracting, according to the preset feature extraction rule, the monitoring feature from the data information corresponding to each user includes: performing word segmentation processing on the data information to obtain a target word segmentation set corresponding to the data information; and filtering the feature participles in the target participle set to obtain the monitoring features, wherein the monitoring features are the feature participles left after the target participle set is filtered and are used for representing the content of the data information.
In an optional implementation manner, before performing feature extraction on the data information, generating a monitoring feature, and monitoring, the method further includes: aggregating the data information according to user dimensions to obtain data information corresponding to each user; and for each user, aggregating the data information corresponding to each user according to a preset time dimension.
In an optional implementation manner, the performing quantity statistics on the monitoring features and determining the target monitoring feature according to a statistical result includes: counting the number of the monitoring features according to a preset counting period aiming at each monitoring feature, and determining the number counting value of the monitoring features; and determining the target monitoring characteristics from the monitoring characteristics according to the quantity statistic value.
In an optional implementation, the determining the target monitoring feature from the monitoring features according to the quantity statistics includes: if the target quantity statistic value in the preset statistic period is smaller than a first quantity threshold value, filtering the monitoring features corresponding to the target quantity statistic value so as to determine the target monitoring features according to the remaining monitoring features.
In an alternative embodiment, the determining the target monitoring characteristic from the monitoring characteristics according to the quantity statistic value includes: acquiring a sum of quantity statistical values corresponding to at least two continuous preset statistical periods; and filtering the monitoring features corresponding to the sum value smaller than a second quantity threshold value so as to determine the target monitoring features according to the remaining monitoring features.
In an optional implementation, after the determining the target monitoring feature from the monitoring features according to the quantity statistics, the method further includes: in the monitoring features, if there are remaining monitoring features of which the number statistic is greater than a third number threshold, the target monitoring feature is updated according to the remaining monitoring features, where the remaining monitoring features are monitoring features other than the target monitoring feature in all the monitoring features.
In an optional implementation manner, the performing a time trend analysis on each target monitoring feature to obtain a feature development trend of each target monitoring feature includes: and aiming at each target monitoring characteristic, generating a time development trend of a quantity statistical value corresponding to the target monitoring characteristic along with the preset statistical period so as to obtain the characteristic development trend.
According to a second aspect of the embodiments of the present disclosure, there is provided a data monitoring apparatus including: the data acquisition module is used for acquiring data information of a data source to be monitored; the characteristic extraction module is used for extracting the characteristics of the data information, generating monitoring characteristics and monitoring; the data processing module is used for carrying out quantity statistics on the monitoring features and determining target monitoring features according to statistical results; and the analysis monitoring module is used for analyzing time trend for each target monitoring characteristic to obtain the characteristic development trend of each target monitoring characteristic so as to carry out monitoring and early warning on the target monitoring characteristics according to the characteristic development trend.
In an alternative embodiment, the analysis monitoring module comprises: the first feature aggregation unit is used for performing feature aggregation on the target monitoring features of which the feature development trends meet preset aggregation conditions; and the aggregation early warning unit is used for taking the characteristic aggregation result as early warning information.
In an alternative embodiment, the feature extraction module comprises: the characteristic extraction unit is used for extracting the monitoring characteristics from the data information corresponding to each user according to a preset characteristic extraction rule; the second feature aggregation unit is used for aggregating the monitoring features of all the users to obtain an aggregated feature set; and the monitoring unit is used for executing monitoring operation on the monitoring features in the aggregation feature set.
In an alternative embodiment, the feature extraction unit comprises: the operation feature determining unit is used for determining the operation features to be searched based on the preset feature extraction rule, wherein the preset feature extraction rule is a rule for searching the operation features of the application program from the data information; the analysis unit is used for carrying out data characteristic analysis on the data information according to the operating characteristics to be searched so as to obtain characteristic result information of the operating characteristics to be searched; and the monitoring feature determining unit is used for taking the operating feature to be searched and the corresponding feature result information as the monitoring feature.
In an alternative embodiment, the feature extraction unit comprises: the word segmentation unit is used for carrying out word segmentation processing on the data information to obtain a target word segmentation set corresponding to the data information; and the word segmentation filtering unit is used for filtering the feature word segmentation in the target word segmentation set to obtain the monitoring feature, and the monitoring feature is the feature word segmentation left after the target word segmentation set is filtered and is used for representing the content of the data information.
In an alternative embodiment, the apparatus further comprises: the first data aggregation module is used for aggregating the data information according to user dimensions to obtain data information corresponding to each user; and the second data aggregation module is used for aggregating the data information corresponding to each user according to a preset time dimension for each user.
In an alternative embodiment, the data processing module comprises: the quantity counting unit is used for counting the quantity of the monitoring features according to a preset counting period aiming at each monitoring feature and determining a quantity counting value of the monitoring features; and the monitoring characteristic determining unit is used for determining the target monitoring characteristic from the monitoring characteristic value according to the quantity statistic value.
In an alternative embodiment, the monitoring feature determination unit includes: the first filtering unit is configured to filter the monitoring features corresponding to the target quantity statistics value if the target quantity statistics value in the preset statistics period is smaller than a first quantity threshold value, so as to determine the target monitoring features according to the remaining monitoring features.
In an alternative embodiment, the monitoring characteristic determining unit includes: the data processing unit is used for acquiring the sum of the quantity statistical values corresponding to at least two continuous preset statistical periods; and the second filtering unit is used for filtering the monitoring features corresponding to the sum smaller than a second quantity threshold value so as to determine the target monitoring features according to the remaining monitoring features.
In an optional implementation, the data processing module further includes: and the updating unit is used for updating the target monitoring feature according to the residual monitoring feature if the residual monitoring feature with the quantity statistic value larger than a third quantity threshold exists in the monitoring features, wherein the residual monitoring feature is the monitoring feature except the target monitoring feature in all the monitoring features.
In an alternative embodiment, the analysis monitoring module is configured to: and aiming at each target monitoring feature, generating a time development trend of a quantity statistic value corresponding to the target monitoring feature along with the preset statistic period so as to obtain the feature development trend.
According to a third aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements any one of the above-mentioned data monitoring methods.
According to a fourth aspect of the disclosed embodiments, there is provided an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the above data monitoring methods via execution of the executable instructions.
According to the data monitoring method, the data monitoring device, the computer-readable storage medium and the electronic device in the embodiments of the disclosure, on one hand, a carving mode of data monitoring is broken through, data information is simplified into monitoring features for monitoring, uniform monitoring without differentiation is performed on data information from different data sources to be monitored, the monitoring features are used as monitoring objects, so that monitoring is not limited by data volume of the data information, and even if the data information with large data volume is a single data, the data monitoring method, the data monitoring device, the computer-readable storage medium and the electronic device have monitoring capability and a wide monitoring range; on the other hand, the target monitoring characteristics are determined according to the number statistical results of the monitoring characteristics, so that monitoring analysis is not limited to preset contents any more, and the data monitoring flexibility is high and the coverage is wide; on the other hand, the data information of the data source to be monitored is continuously updated, the extracted monitoring characteristics and the determined target monitoring characteristics are dynamically adjusted, so that the content of data monitoring is always consistent with the actual condition of the data source to be monitored, and the real-time effectiveness of data monitoring is improved; in addition, target monitoring features are screened as analysis objects through the number statistical results of the monitoring features, the data analysis range is further positioned, service resources of a monitoring stage and a analysis stage in the data monitoring process are coordinated, monitoring early warning misinformation is further avoided, monitoring early warning accuracy is improved, and the processing efficiency of follow-up problem services is improved.
Drawings
The above and other objects, features and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description read in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:
FIG. 1 shows a flow diagram of a data monitoring method according to an embodiment of the present disclosure;
FIG. 2 illustrates a time trend analysis plot of quantity statistics over time for a target monitoring feature M according to an embodiment of the present disclosure;
FIG. 3 shows a comparison diagram of feature development trends for target monitoring features according to an embodiment of the present disclosure;
FIG. 4 illustrates a temporal trend analysis graph with anomalous features according to an embodiment of the present disclosure;
FIG. 5 illustrates a flow diagram of a method of extracting monitor features according to an embodiment of the present disclosure;
FIG. 6 shows a schematic diagram of extracting monitoring features for each user according to an embodiment of the present disclosure;
FIG. 7 shows a schematic diagram of aggregating data information, according to an embodiment of the present disclosure;
FIG. 8 illustrates a schematic diagram of an aggregate monitoring feature according to an embodiment of the present disclosure;
FIG. 9 illustrates a flow diagram for extracting monitoring features from data information in accordance with an embodiment of the present disclosure;
FIG. 10 shows a schematic diagram of a data monitoring device according to an embodiment of the present disclosure;
FIG. 11 shows a schematic diagram of a storage medium according to an embodiment of the present disclosure;
FIG. 12 shows a schematic diagram of an electronic device according to an embodiment of the disclosure.
Detailed Description
The principles and spirit of the present disclosure will be described with reference to a number of exemplary embodiments. It is understood that these embodiments are given solely for the purpose of enabling those skilled in the art to better understand and to practice the present disclosure, and are not intended to limit the scope of the present disclosure in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
As will be appreciated by one skilled in the art, embodiments of the present disclosure may be embodied as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the form of: entirely hardware, entirely software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.
According to an embodiment of the present disclosure, a data monitoring method, a data monitoring apparatus, a computer-readable storage medium, and an electronic device are provided.
In this document, any number of elements in the drawings is by way of example and not by way of limitation, and any nomenclature is used solely for differentiation and not by way of limitation.
The principles and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.
Summary of The Invention
The existing data monitoring method monitors, discovers problems, warns and processes a data source to be monitored according to preset indexes, but the monitoring range of the method is limited to the content corresponding to the preset indexes, only specific problems can be discovered, and aiming at other indexes which are not preset, the method has no monitoring capability and is difficult to discover unknown problems; in addition, when the data volume of the data information exceeds a certain threshold range, the existing method cannot monitor and analyze a large amount of data information and give an early warning in time. In the embodiment of the disclosure, the data information of various data sources to be monitored is simplified into the monitoring characteristics, and the monitoring is not limited by the data volume of the data information as the monitoring object, so that even the data information with large data volume is single, the monitoring capability is provided, and the monitoring range is wide. In order to avoid the limitation of monitoring analysis to preset contents, the target monitoring characteristics are determined according to the number statistical results of the monitoring characteristics, unknown problems can be found in time, and the data monitoring coverage is wide. And in the process of continuously updating the data source of the monitoring data, the extracted monitoring characteristics and the determined target monitoring characteristics are also adjusted, and the monitored object has real-time effectiveness.
Having described the general principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.
Exemplary application scenarios
It should be noted that the following application scenarios are merely illustrated to facilitate understanding of the spirit and principles of the present disclosure, and embodiments of the present disclosure are not limited in this respect. Rather, embodiments of the present disclosure may be applied to any scenario where applicable.
The data monitoring method of the embodiment of the disclosure can be applied to various application scenarios related to data monitoring.
In one Application scenario, APP (Application) operation monitoring may be involved. Generally, in such an application scenario, the user terminal is pre-installed with APPs and third party APPs installed by the user, such as music, games, navigation, and other software. During the period of using the APP by the user, abnormalities may be caused by the terminal, the network, the APP version and the like, and the normal use of the APP is influenced. By using the data monitoring method of the embodiment of the disclosure, data information of each data source to be monitored during the running period of the APP can be obtained first, for example, the data information is obtained from the data source to be monitored, such as APP logs, server logs and/or customer complaint communication texts fed back by users through customer service channels, then the monitoring features are extracted from the data information and monitored, the target monitoring features are determined according to the data statistical results of the monitoring features, time trend analysis is performed, and monitoring and early warning are performed on the target monitoring features according to the time trend analysis results.
In another application scenario, Web (World Wide Web) service platform monitoring may be involved. Generally, in such an application scenario, a target system is monitored for a service interface, and a web service is exposed to the outside by calling the target system, but it has no way to know whether the web service exposed to the outside is normal or not. By using the data monitoring method of the embodiment of the disclosure, data information from different data sources to be monitored during the operation of the web service platform is collected, the monitoring characteristics are extracted for monitoring, and the target monitoring characteristics are determined for time trend analysis, so that the characteristic granularity can be monitored for various operation data generated during the operation of the web service platform, and the problem that the operation data cannot provide services for a long time but cannot be discovered is avoided.
Exemplary method
In conjunction with the above application scenarios, a data monitoring method according to an exemplary embodiment of the present disclosure is described below with reference to fig. 1.
Fig. 1 shows a flowchart of a data monitoring method according to an exemplary embodiment of the present disclosure, which may include steps S110 to S140:
step S110, data information of a data source to be monitored is obtained.
In an exemplary embodiment of the present disclosure, the data source to be monitored is a channel for recording various events occurring during the running of the application, for example, in the application scenario of APP running monitoring described above, the data source to be monitored includes, but is not limited to, an APP log that records events occurring during the running of the APP and is stored locally, a server log that is recorded by the server and is stored in a file created by the server when the user uses the APP, a customer feedback complaint that is fed back to the APP developer by the user through a customer service channel, and the like. The exemplary embodiment of the present disclosure does not make any special limitation on the type of the data source to be monitored, and all channels in which various events occur during the application running period are recorded can be used as the data source to be monitored.
The formats of the data information corresponding to different data sources to be monitored can be various, such as plain text and log source codes, and the format of the data information is not specially limited in the embodiment of the disclosure.
The server side can actively send a log obtaining request to the terminal where the application is located and receive an APP log returned by the terminal where the application is located; or, the terminal where the application is located may also actively upload the APP log stored locally to the time-lapse server; or, the terminal where the application is located uploads the APP log stored in the local area to the specified memory space, for example, the file server, and the server can obtain the APP log from the specified memory space, so that not only can the APP log be prevented from occupying the local memory, but also the server can conveniently obtain the APP log from the specified memory space.
The data information of different data sources to be monitored is obtained, and the coverage range of data monitoring is increased, so that effective monitoring characteristics can be extracted from the data information of all the data sources to be monitored in the following process, and the missing of abnormal problem items is avoided.
In step S120, feature extraction is performed on the data information, and a monitoring feature is generated and monitored.
In the exemplary embodiment of the present disclosure, the feature extraction of the data information is to extract a feature result reflecting the content of the data information from the data information. The data information of different data sources to be monitored may have different data types, and for the data sources to be monitored of different data types, the embodiment of the present disclosure further provides a corresponding feature extraction rule, and based on a preset feature extraction rule, standardized monitoring features are extracted from the data information belonging to different data types, so that the data information of different data sources to be monitored can be subjected to type-indifferent monitoring features.
For example, if the data information 1 of the data source 1 to be monitored is obtained as "probability can not enter into the cardiac model when local music is collected", the extracted monitoring features are "collection", "local", "music", "probability", "appearance", "can not", "enter", "cardiac mode" based on the feature extraction rule corresponding to the data type 1, the data information 2 of the data source 2 to be monitored is a section of log source code "private String ID … … private String app ID", the extracted monitoring features are "enterprise user" and "application identification" based on the feature extraction rule corresponding to the data type 2, and then the data information of different data sources to be monitored is extracted as the monitoring features with no type difference.
If the data information of the data source to be monitored changes, such as addition, modification, deletion and the like, the monitoring characteristics extracted from the data information are correspondingly dynamically adjusted, so that the monitoring characteristics are always consistent with the actual conditions of the data source to be monitored, and the real-time effectiveness of data monitoring is improved.
In some possible embodiments, before performing the feature extraction on the data information, the source of the data information may also be determined, and the corresponding feature extraction rule is selected according to the source of the actual information. Optionally, when the data information of the data source to be monitored is acquired, a tag identifying the source of the data information may be added to the data information, and then the target feature extraction rule required by the data information is determined based on the corresponding relationship between the tag and the preset feature extraction rule. Based on this, the data information of different sources is convenient to be integrated and managed.
Through the exemplary embodiment of the disclosure, data information is simplified into monitoring characteristics and serves as a monitoring object, invalid information is filtered, so that data monitoring is not limited by the data volume of the data information any more, and the data information has monitoring capability even if the data information with large data volume exists; in addition, data information of all data sources to be monitored is simplified into monitoring characteristics with standardized formats, integration and unified monitoring of the monitoring characteristics of all data sources to be monitored are facilitated, and data monitoring across data sources is achieved.
In step S130, the number of the monitoring features is counted, and the target monitoring feature is determined according to the counting result.
In the exemplary embodiment of the disclosure, after the data information of each data source to be monitored is simplified into the fine-grained monitoring features, the target monitoring features may be determined from the monitoring features according to the number statistical result of the monitoring features, and the target monitoring features are used as subsequent data analysis objects. The quantity statistic value of each monitoring feature can be counted according to a preset counting period, and the target monitoring feature is determined from the monitoring features according to the quantity statistic value. The preset statistical period may be one day, two days, five days, and the like, and the corresponding statistical period may be selected according to the actual data monitoring requirement, which is not particularly limited in the embodiment of the present disclosure.
The target monitoring characteristics can be screened from the monitoring characteristics according to the size of the quantity statistic value, and the target monitoring characteristics can be screened from the monitoring characteristics according to the distribution rule of the monitoring characteristics along with time. For example, if the number statistic value of the monitoring feature M in the preset statistical period does not reach the threshold value, but the monitoring feature M appears every day in the preset statistical period, which indicates that the problem item corresponding to the monitoring feature may exist and is not solved, the monitoring feature M is also used as the target monitoring feature. Of course, the specific implementation manner of determining the target monitoring features from the monitoring features may also be adjusted according to the actual data monitoring needs, and the present disclosure includes, but is not limited to, the above method of screening the target monitoring features from the monitoring features.
The target monitoring features are determined according to the number statistical results of the monitoring features, so that monitoring analysis is not limited to preset contents any more, the data analysis flexibility is high, the coverage is wide, the data analysis range is further positioned, service resources of a monitoring stage and an analysis stage in the data monitoring process are coordinated, the monitoring features with errors introduced by feature extraction can be avoided, the monitoring early warning misinformation is avoided, and the monitoring early warning accuracy is improved.
In step S140, time trend analysis is performed on each target monitoring feature to obtain a feature development trend of each target monitoring feature, so as to perform monitoring and early warning on the target monitoring feature according to the feature development trend.
According to an exemplary embodiment of the present disclosure, the time trend analysis is to analyze a transformation rule of the target monitoring feature over time. As shown in fig. 2, a time trend analysis graph of the number statistics of the target monitoring features M changing with time is shown, and if there is a trend that needs to trigger an early warning in the time trend analysis shown in fig. 2, such as a situation of sudden rise, sudden fall, severe fluctuation, stability at a certain value, and the like, the early warning may be triggered. The triggering of the early warning may be sending out an early warning of "abnormal problem occurs and continues to occur", or sending out an early warning of "reminding abnormal problem and problem solved", or sending out an early warning of "reminding abnormal problem occurs and there is a risk of causing some new problems", or the like.
Optionally, for each target monitoring feature, a corresponding feature development trend may be generated according to the same time trend. As shown in fig. 3, for a target monitoring feature "region: city "and" network: the education network can respectively generate respective characteristic development trends by taking the number statistics of the two as a unit of day; optionally, for the target monitoring features, corresponding feature development trends may also be generated according to different time trends, the feature development trends may be generated by taking the number statistics as a unit for 1 day, the feature development trends may also be generated by taking the number statistics as a unit for 2 days, and of course, the corresponding time trends may be selected in consideration of actual monitoring requirements of different target monitoring features.
The target monitoring features serving as key analysis objects are determined according to the number statistical results of the monitoring features, unknown abnormal problems can be identified by analyzing the time trend of the target monitoring features, and the data monitoring has high flexibility and is comprehensive.
By the data monitoring method, a carving mode of data monitoring is broken through, data information is simplified into monitoring characteristics for monitoring, uniform monitoring without differentiation is carried out on the data information from different data sources to be monitored, and monitoring is not limited by the data volume of the data information; the target monitoring characteristics are determined according to the number statistical results of the monitoring characteristics, so that monitoring analysis is not limited to preset contents any more, and the data monitoring flexibility is high and the coverage is wide; the data information of the data source to be monitored is continuously updated, the extracted monitoring characteristics and the determined target monitoring characteristics are dynamically adjusted, so that the content of data monitoring is always consistent with the actual condition of the data source to be monitored, and the real-time effectiveness of data monitoring is improved; in addition, target monitoring features are screened as analysis objects through the number statistical results of the monitoring features, the data analysis range is further positioned, service resources of a monitoring stage and a analysis stage in the data monitoring process are coordinated, monitoring early warning misinformation is further avoided, monitoring early warning accuracy is improved, and the processing efficiency of follow-up problem services is improved.
In an exemplary embodiment of the present disclosure, a method for generating a feature development trend is also provided. Performing time trend analysis on each target monitoring feature to obtain a feature development trend of each target monitoring feature may be: and aiming at each target monitoring characteristic, generating a time development trend of a quantity statistical value corresponding to the target monitoring characteristic along a preset statistical period so as to obtain a characteristic development trend.
For example, if the preset statistical period is 1 day, the number of statistics of the target monitoring features develops over time, which is 1, 2, 1, 0, and 2, respectively, the characteristic development trend is that each number of statistics forms a time development trend according to a distribution rule corresponding to the day, and as shown in fig. 3, the "region: city "and" network: the education network takes days as a preset statistical period to generate a characteristic development trend of the target monitoring characteristics. Of course, if the preset statistical period is one week, one quantity statistical value is the total quantity value of the target monitoring feature in one week, and the feature development trend is a feature development trend of the target monitoring feature along with time according to the distribution rule corresponding to the week of each quantity statistical value.
For example, with continued reference to fig. 3, on day 1, month 8 of 2022, the number statistics of the target monitoring features 1 is 2, the number statistics of the target monitoring features 2 is 1, and so on, the number statistics of the target monitoring features on days 1, month 9 of 2022, month 1, month 10 of 2022, and so on can be obtained. On the basis, the characteristic development trend of each target monitoring characteristic in days can be generated, and correspondingly, the quantity statistics values in adjacent days can be aggregated, and the change trend of the aggregation result along with time can be generated.
In an exemplary embodiment of the present disclosure, an implementation of the aggregate monitoring and forewarning is also provided. According to the characteristic development trend of the target monitoring characteristics, monitoring and early warning are carried out on the target monitoring characteristics, and the method can comprise the following steps:
performing feature aggregation on a plurality of target monitoring features of which the feature development trends meet preset aggregation conditions; and taking the feature aggregation result as early warning information.
Alternatively, the preset polymerization conditions include: at the same time, the variation trend of the quantity statistical value of the target monitoring characteristic is abnormal and is the same. The anomaly may be that the magnitude of the quantity statistics exceeds a magnitude threshold, that the magnitude of the quantity statistics falls exceeds a magnitude threshold, and that the fluctuation frequency of the quantity statistics exceeds a frequency threshold. As shown in fig. 4, the target monitoring feature "region: city D and network: the quantity statistics value of the education network is abnormally increased (the increase of the quantity statistics value exceeds the increase threshold value by 30 percent) at the same time, and the increase amplitudes of the quantity statistics value and the increase amplitude are the same, so that the preset aggregation condition is met.
Alternatively, the preset polymerization conditions include: and in the target monitoring characteristics before and after the preset time period, the variation trend of the quantity statistical value is abnormal and is the same. For example, the target monitoring feature "zone: city D's number statistics are growing abnormally at 12:00, while target monitoring features' network: the educational network "also grows abnormally at 12:05, and both grow the same, which may be the target monitoring feature" territory: city D "cause the target to monitor the characteristic" network abnormally: if the education network is abnormal, the education network and the education network accord with the preset aggregation condition. Of course, the preset time period of 5 minutes is only exemplary, and the preset time period of the embodiment of the present disclosure may also be set according to the actual data monitoring requirement.
Optionally, the preset polymerization conditions include: and in the target monitoring characteristics at the same time or before and after the preset time period, the variation trend of the quantity statistical value is abnormal and is opposite. For example, the target monitoring feature "zone: city "and" network: the quantity statistics value of the education network is abnormal at the same time, the change range of the quantity statistics value is the same as that of the quantity statistics value, but the trend is opposite, and the preset aggregation condition is met. Specifically, "region: city D "the fluctuation of the quantity statistics exceeds the fluctuation threshold by 30%," network: the drop of the quantity statistics value of the education network is 30% above the drop threshold, the preset aggregation condition is met, in this case, the two may have mutual influence, and therefore the two can be used as exception eliminating factors.
In some possible embodiments, a variation range interval of a plurality of quantity statistics values may be preset, a variation range interval to which the target monitoring feature belongs may be determined according to the quantity statistics value of the target monitoring feature, and a plurality of target monitoring features belonging to the same variation range interval may be aggregated. For example, the variation range interval may include [0, 5% ], [ 5%, 10% ], [ 10%, 15% ], [ 15%, 30% ] and so on, and the embodiments of the present disclosure may set different variation range intervals according to actual monitoring requirements, which is not particularly limited by the present disclosure.
The method comprises the steps of performing feature aggregation on a plurality of target monitoring features meeting preset aggregation conditions, using feature aggregation results as early warning information, correlating various abnormal information, and providing auxiliary reference for rapid early warning or abnormal positioning. For example, let "region: city "and" network: the education network is simultaneously used as early warning information, and can help to confirm that the education network in the city D possibly has network faults, so that the data information from the data source to be monitored includes' region: city "and" network: education network "abnormal changes occurred simultaneously.
In an exemplary embodiment of the present disclosure, a method of extracting a monitoring feature is also provided. The performing feature extraction on the data information, generating a monitoring feature and monitoring may include steps S510 to S530:
step S510, extracting a monitoring feature from the data information corresponding to each user according to a preset feature extraction rule.
And extracting the monitoring characteristics according to a preset characteristic extraction rule aiming at the data information corresponding to each user. As shown in fig. 6, a schematic diagram of extracting monitoring features of a certain user according to an exemplary embodiment of the present disclosure includes 283 user logs on a certain day for a certain user, a plurality of monitoring features (circle marks) are respectively extracted from each user log, and all the monitoring features of the user are obtained by aggregating the extracted plurality of monitoring features. Accordingly, the monitoring characteristics of the user every day can be obtained.
In an exemplary embodiment of the present disclosure, if the acquired data information of the data source to be monitored is not stored in a user unit, the data information may also be aggregated according to a user dimension to obtain data information corresponding to each user, and for each user, the data information corresponding to each user is aggregated according to a preset time dimension. That is, the data information may be aggregated according to the user, and then, for each user, the data information may be aggregated according to the preset time dimension, as shown in fig. 7, and the data information is aggregated according to the user dimension and the preset time dimension (one day).
The data information is aggregated according to the user dimension, and the data information corresponding to each user is aggregated according to the preset time dimension, so that the data information is convenient to count, and the storage mode of the application running data is also met.
Step S520, performing aggregation processing on the monitoring features of all users to obtain an aggregation feature set.
After the data information of each data source to be monitored is simplified into fine-grained monitoring features, the monitoring features of all users are aggregated to obtain an aggregated feature set, and the aggregated feature set is used as a monitoring object. As shown in fig. 8, the monitoring features from the three data sources to be monitored are respectively output, the monitoring features of all the users from all the data sources to be monitored are aggregated, and the formed aggregation feature set includes all the relevant information applied during the use period of the users, so that the application is monitored in all directions.
Step S530, performing a monitoring operation on the monitoring features in the aggregated feature set.
And after the aggregation feature set is obtained, monitoring operation is performed on the monitoring features in the monitoring feature set, the monitoring features are used as monitoring objects, and undifferentiated unified monitoring is performed on data information from different data sources to be monitored, so that the monitoring range is wide.
In an exemplary embodiment of the present disclosure, an implementation of extracting monitoring features from data information is also provided. Extracting the monitoring feature from the data information corresponding to each user according to a preset feature extraction rule, which may include steps S910 to S930:
step S910, determining an operation feature to be searched based on a preset feature extraction rule.
The preset feature extraction rule is a rule for searching the running features of the application program from the data information, the running features include but are not limited to network environment, network speed, application version, system type, region, application state, system language and other relevant features during application running, and such data information may be log source code or other data information written by machine language. The preset feature extraction rule comprises a search logic compiled based on the operation features to be searched and used for processing the data information and analyzing the operation features to be searched, and the operation features to be searched can be determined firstly according to the search logic. Alternatively, the preset feature extraction rule may be an analyzer written based on the to-be-searched operating feature, such as a network environment analyzer, the determined to-be-searched operating feature is a network environment, the to-be-searched operating feature determined by the version analyzer is a version type, and the system determines that the to-be-searched operating feature determined by the analyzer is a system language.
In some possible embodiments, the parser may expand the feature parsing capability according to the data monitoring requirement, and perform function iteration according to the implementation code. When the resolver is updated, such as function iteration, addition, deletion, and the like, the embodiment of the disclosure may also determine the corresponding operating characteristic to be searched according to the updated resolver. Based on the method, the accuracy of the determined running characteristic to be searched can be improved in the operation processes of continuous updating iteration, new addition and deletion.
Step S920, according to the operation feature to be searched, performing data feature analysis on the data information to obtain feature result information of the operation feature to be searched.
And based on the operation characteristics to be searched, corresponding searching logic is operated on the data information to analyze the data characteristics of the data information, so that characteristic result information of the operation characteristics to be searched is obtained. The characteristic result information is actual data information corresponding to the operation characteristic to be searched in the data information, for example, if the operation characteristic to be searched is a network environment, based on the network environment, the downloading speed is searched in the data information, and the network speed state (such as network speed slow, network speed medium, network speed fast and network disconnection) is determined according to the downloading speed, and the network speed state is used as the characteristic result information of the network environment.
In some possible embodiments, a plurality of resolvers may also be used simultaneously, and accordingly, a plurality of to-be-searched operation features may be determined simultaneously, and data feature analysis may be performed on the data information according to the to-be-searched operation features to obtain a plurality of corresponding feature result information. For example, a plurality of feature result information can be searched at the same time by using a network speed state as feature result information of a network environment, a region state as feature result information, a network type as feature result information, an education network as feature result information, and the like.
Step S930, using the operation feature to be searched and the corresponding feature result information as the monitoring feature.
A monitoring feature includes an operational feature to be looked up and corresponding feature result information, such as network environment: network disconnection and software version: bb01, system language: chinese, and so on.
If multiple pieces of feature result information are found at the same time in step S920, multiple pairs of "operation feature to be found — feature result information" may be obtained at the same time in step S930, so as to obtain a finer analysis result of the data information.
In an exemplary embodiment of the present disclosure, another implementation of extracting monitoring features from data information is also provided. Extracting the monitoring features from the data information corresponding to each user according to a preset feature extraction rule, which may include:
performing word segmentation processing on the data information to obtain a target word segmentation set corresponding to the data information;
and filtering the characteristic participles in the target participle set to obtain monitoring characteristics, wherein the monitoring characteristics are the characteristic participles left after the target participle set is filtered and are used for representing the content of the data information.
The data information may be plain text, such as a communication text of the user for feeding back the customer complaint. For such data information, word segmentation processing may be performed on the data information, and the word segmentation method includes, but is not limited to, a dictionary-based word segmentation method (such as forward Maximum Matching algorithm MM (Maximum Matching)), reverse Maximum Matching algorithm rmm (reverse Maximum Matching), bidirectional Maximum Matching algorithm BM (Bi-direction Maximum Matching)), a statistic-based word segmentation method (such as N-gram model), and the like, and this is not particularly limited in the embodiment of the present disclosure.
Furthermore, the participles in the target participle set are not all effective participle results, such as auxiliary words and conjunctions like "when", "after", etc., and may have no practical meaning to the content of the reflected data information, such participles are filtered; alternatively, synonyms existing in the target set of participles may also be filtered, such as: "fail" and "fail", "present" and "present"; or repeated word segmentation in the data information can be filtered, so that the final monitoring characteristics can accurately reflect the content of the data information without redundancy and ambiguity.
In some possible embodiments, a filtering blacklist may be preset, and the target segmented word set is filtered according to the filtering blacklist, so as to improve the filtering processing efficiency.
In an exemplary embodiment of the present disclosure, an implementation of determining target monitoring characteristics is also provided. Determining the target monitoring feature from the monitoring features according to the quantity statistics may include: and if the target quantity statistic value in the preset statistic period is smaller than the first quantity threshold value, filtering the monitoring features corresponding to the target quantity statistic value to determine the target monitoring features according to the remaining monitoring features.
The preset statistical period may be one day, several days, one week, etc., and this disclosure is not limited thereto. For example, monitoring features that occur less than 10 times per day may be filtered out; as another example, monitoring features that total occurrences of less than 100 for five days may be filtered out. Based on the data analysis method, the data analysis range is further positioned, service resources of a monitoring stage and a subsequent analysis stage in the data monitoring process are coordinated, meanwhile, the data analysis range is positioned to a target monitoring characteristic appearing at a higher frequency, and the data monitoring accuracy is improved.
In an exemplary embodiment of the present disclosure, another implementation of determining target monitoring characteristics is also provided. Determining the target monitoring feature from the monitoring features according to the quantity statistics may include: acquiring a sum of quantity statistics corresponding to at least two continuous preset statistics periods; and filtering the monitoring features corresponding to the sum value smaller than the second quantity threshold value so as to determine the target monitoring features according to the remaining monitoring features. Based on this, although in a preset statistical period (for example, 1 day), the quantity statistical value of a certain monitoring feature reaches the first quantity threshold value, in three consecutive preset statistical periods (for example, 3 days), the sum of the quantity statistical values of the monitoring feature does not reach the second quantity threshold value, and the monitoring feature is filtered out, so that the interference feature exists in the monitoring feature, and the monitoring early warning accuracy is improved.
In an exemplary embodiment of the present disclosure, an implementation of updating target monitoring features is also provided. After the target monitoring feature is determined from the monitoring features according to the quantity statistics, if there are remaining monitoring features of which the quantity statistics is greater than the third quantity threshold in the monitoring features, the target monitoring feature may be updated according to the remaining monitoring features.
And the rest monitoring features are monitoring features except the target monitoring feature in all the monitoring features. For example, the monitoring features include [ m, n, q, e, f, g ], the target monitoring feature determined for the first time is [ m, n, q, e ], the remaining monitoring features are [ f, g ], in the process of continuous monitoring, data information of the data source to be monitored is continuously updated, the extracted monitoring features are dynamically adjusted accordingly, the quantity statistical value corresponding to the monitoring features also changes, if the quantity statistical value of the remaining monitoring features f is greater than a third quantity threshold value, the remaining monitoring features f can be added into the target monitoring feature set to obtain [ m, n, q, e, f ], so that the content of data monitoring is consistent with the actual situation of the data source to be monitored all the time, and the real-time effectiveness of data monitoring is improved.
Exemplary devices
Having described the data monitoring method of the exemplary embodiment of the present disclosure, next, a data monitoring apparatus of the exemplary embodiment of the present disclosure will be explained with reference to fig. 10.
Fig. 10 shows a data monitoring apparatus 1000 of an exemplary embodiment of the present disclosure, including:
the data acquisition module 1010 is used for acquiring data information of a data source to be monitored;
a feature extraction module 1020, configured to perform feature extraction on the data information, generate a monitoring feature, and perform monitoring;
the data processing module 1030 is configured to perform quantity statistics on the monitoring features, and determine target monitoring features according to a statistical result;
the analyzing and monitoring module 1040 is configured to perform time trend analysis on each target monitoring feature to obtain a feature development trend of each target monitoring feature, and perform monitoring and early warning on the target monitoring feature according to the feature development trend.
In an alternative embodiment, the analysis monitoring module comprises:
the first feature aggregation unit is used for performing feature aggregation on the target monitoring features of which the feature development trends meet preset aggregation conditions; and the aggregation early warning unit is used for taking the feature aggregation result as early warning information.
In an alternative embodiment, the feature extraction module comprises:
the characteristic extraction unit is used for extracting the monitoring characteristics from the data information corresponding to each user according to a preset characteristic extraction rule;
the second feature aggregation unit is used for aggregating the monitoring features of all the users to obtain an aggregated feature set;
and the monitoring unit is used for executing monitoring operation on the monitoring features in the aggregation feature set.
In an alternative embodiment, the feature extraction unit comprises:
the operation feature determining unit is used for determining the operation features to be searched based on the preset feature extraction rule, wherein the preset feature extraction rule is a rule for searching the operation features of the application program from the data information;
the analysis unit is used for carrying out data characteristic analysis on the data information according to the operating characteristics to be searched so as to obtain characteristic result information of the operating characteristics to be searched;
and the monitoring characteristic determining unit is used for taking the operating characteristic to be searched and the corresponding characteristic result information as the monitoring characteristic.
In an alternative embodiment, the feature extraction unit comprises:
the word segmentation unit is used for carrying out word segmentation processing on the data information to obtain a target word segmentation set corresponding to the data information;
and the word segmentation filtering unit is used for filtering the feature word segmentation in the target word segmentation set to obtain the monitoring feature, and the monitoring feature is the feature word segmentation left after the target word segmentation set is filtered and is used for representing the content of the data information.
In an alternative embodiment, the apparatus further comprises:
the first data aggregation module is used for aggregating the data information according to user dimensions to obtain data information corresponding to each user;
and the second data aggregation module is used for aggregating the data information corresponding to each user according to a preset time dimension for each user.
In an alternative embodiment, the data processing module comprises:
the quantity counting unit is used for counting the quantity of the monitoring features according to a preset counting period aiming at each monitoring feature and determining a quantity counting value of the monitoring features;
and the monitoring characteristic determining unit is used for determining the target monitoring characteristic from the monitoring characteristic value according to the quantity statistic value.
In an alternative embodiment, the monitoring feature determination unit includes:
the first filtering unit is configured to filter the monitoring features corresponding to the target quantity statistics value if the target quantity statistics value in the preset statistics period is smaller than a first quantity threshold value, so as to determine the target monitoring features according to the remaining monitoring features.
In an alternative embodiment, the monitoring feature determination unit includes:
the data processing unit is used for acquiring the sum of the quantity statistical values corresponding to at least two continuous preset statistical periods;
and the second filtering unit is used for filtering the monitoring features corresponding to the sum smaller than a second quantity threshold value so as to determine the target monitoring features according to the remaining monitoring features.
In an optional implementation, the data processing module further includes:
and the updating unit is used for updating the target monitoring feature according to the residual monitoring feature if the residual monitoring feature with the quantity statistic value larger than a third quantity threshold exists in the monitoring features, wherein the residual monitoring feature is the monitoring feature except the target monitoring feature in all the monitoring features.
In an alternative embodiment, the analysis monitoring module is configured to:
and aiming at each target monitoring characteristic, generating a time development trend of a quantity statistical value corresponding to the target monitoring characteristic along with the preset statistical period so as to obtain the characteristic development trend.
It should be noted that other specific details of each functional module of the data monitoring apparatus according to the embodiment of the present disclosure have been described in detail in the embodiment of the data monitoring method, and are not described herein again.
It should be noted that although in the above detailed description several modules or units of the data monitoring device are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
Exemplary storage Medium
The storage medium of the exemplary embodiment of the present disclosure is explained below.
In this exemplary embodiment, referring to fig. 11, a program product 1100 for implementing the above-described method according to an exemplary embodiment of the present disclosure is described, such as may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RE, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (FAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
Exemplary electronic device
An electronic device of an exemplary embodiment of the present disclosure is explained with reference to fig. 12.
The electronic device 1200 shown in fig. 12 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 12, electronic device 1200 is embodied in the form of a general-purpose computing device. The components of the electronic device 1200 may include, but are not limited to: at least one processing unit 1210, at least one memory unit 1220, a bus 1230 connecting the various system components including the memory unit 1220 and the processing unit 1210, and a display unit 1240.
Where the memory unit stores program code, the program code may be executed by the processing unit 1210 such that the processing unit 1210 performs the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "exemplary methods" section of this specification. For example, processing unit 1210 may perform the method steps, etc., shown in fig. 1.
The storage unit 1220 may include volatile storage units such as a random access memory unit (RAM)1221 and/or a cache memory unit 1222, and may further include a read only memory unit (ROM) 1223.
Storage unit 1220 may also include programs/utilities 1224 having a set (at least one) of program modules 1225, such program modules 1225 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The bus 1230 may include a data bus, an address bus, and a control bus.
The electronic device 1200 may also communicate with one or more external devices 1300 (e.g., keyboard, pointing device, bluetooth device, etc.) via an input/output (I/O) interface 1250. The electronic device 1200 further comprises a display unit 1240 connected to the input/output (I/O) interface 1250 for displaying. Also, the electronic device 1200 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 1260. As shown, the network adapter 1260 communicates with the other modules of the electronic device 1200 via the bus 1230. It should be appreciated that although not shown, other hardware and/or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
It should be noted that although in the above detailed description several modules or sub-modules of the apparatus are mentioned, such division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the units/modules described above may be embodied in one unit/module, in accordance with embodiments of the present disclosure. Conversely, the features and functions of one unit/module described above may be further divided into embodiments by a plurality of units/modules.
Further, while the operations of the disclosed methods are depicted in the drawings in a particular order, this does not require or imply that these operations must be performed in this particular order, or that all of the illustrated operations must be performed, to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
While the spirit and principles of the present disclosure have been described with reference to several particular embodiments, it is to be understood that the present disclosure is not limited to the particular embodiments disclosed, nor is the division of aspects, which is for convenience only as the features in such aspects may not be combined to benefit. The disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims (10)

1. A method for monitoring data, comprising:
acquiring data information of a data source to be monitored;
performing feature extraction on the data information to generate monitoring features and monitoring;
counting the number of the monitoring features, and determining target monitoring features according to the counting result;
and analyzing time trend for each target monitoring feature to obtain the feature development trend of each target monitoring feature, and monitoring and early warning the target monitoring features according to the feature development trend.
2. The method of claim 1, wherein the monitoring and early warning the target monitoring feature according to the feature development trend comprises:
and performing feature aggregation on a plurality of target monitoring features of which the feature development trends meet preset aggregation conditions, and taking feature aggregation results as early warning information.
3. The method according to claim 1, wherein the performing feature extraction on the data information, generating a monitoring feature and monitoring includes:
extracting the monitoring features from the data information corresponding to each user according to a preset feature extraction rule;
aggregating the monitoring characteristics of all users to obtain an aggregated characteristic set;
performing a monitoring operation on the monitoring features in the aggregated feature set.
4. The method according to claim 3, wherein the extracting the monitoring feature from the data information corresponding to each user according to the preset feature extraction rule comprises:
determining an operation feature to be searched based on the preset feature extraction rule, wherein the preset feature extraction rule is a rule for searching the operation feature of the application program from the data information;
according to the operating characteristics to be searched, carrying out data characteristic analysis on the data information to obtain characteristic result information of the operating characteristics to be searched;
and taking the operating characteristics to be searched and the corresponding characteristic result information as the monitoring characteristics.
5. The method according to claim 3, wherein the extracting the monitoring feature from the data information corresponding to each user according to the preset feature extraction rule includes:
performing word segmentation processing on the data information to obtain a target word segmentation set corresponding to the data information;
and filtering the feature participles in the target participle set to obtain the monitoring features, wherein the monitoring features are the feature participles left after the target participle set is filtered and are used for representing the content of the data information.
6. The method of claim 1, wherein the counting the number of the monitoring features and determining the target monitoring feature according to the counting result comprises:
counting the number of the monitoring features according to a preset counting period aiming at each monitoring feature, and determining the number counting value of the monitoring features;
and determining the target monitoring characteristics from the monitoring characteristics according to the quantity statistic value.
7. The method of claim 6, wherein after said determining said target monitoring feature from said monitoring features according to said quantity statistics, said method further comprises:
in the monitoring features, if there are remaining monitoring features of which the number statistic is greater than a third number threshold, the target monitoring feature is updated according to the remaining monitoring features, where the remaining monitoring features are monitoring features other than the target monitoring feature in all the monitoring features.
8. A data monitoring apparatus, the apparatus comprising:
the data acquisition module is used for acquiring data information of a data source to be monitored;
the characteristic extraction module is used for extracting the characteristics of the data information, generating monitoring characteristics and monitoring;
the data processing module is used for carrying out quantity statistics on the monitoring features and determining target monitoring features according to statistical results;
and the analysis monitoring module is used for analyzing time trend for each target monitoring characteristic to obtain the characteristic development trend of each target monitoring characteristic so as to carry out monitoring and early warning on the target monitoring characteristics according to the characteristic development trend.
9. A storage medium having stored thereon a computer program which, when executed by a processor, implements a data monitoring method according to any one of claims 1 to 7.
10. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the data monitoring method of any one of claims 1 to 7 via execution of the executable instructions.
CN202210219898.6A 2022-03-08 2022-03-08 Data monitoring method and device, storage medium and electronic equipment Pending CN114595118A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210219898.6A CN114595118A (en) 2022-03-08 2022-03-08 Data monitoring method and device, storage medium and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210219898.6A CN114595118A (en) 2022-03-08 2022-03-08 Data monitoring method and device, storage medium and electronic equipment

Publications (1)

Publication Number Publication Date
CN114595118A true CN114595118A (en) 2022-06-07

Family

ID=81816040

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210219898.6A Pending CN114595118A (en) 2022-03-08 2022-03-08 Data monitoring method and device, storage medium and electronic equipment

Country Status (1)

Country Link
CN (1) CN114595118A (en)

Similar Documents

Publication Publication Date Title
US11586972B2 (en) Tool-specific alerting rules based on abnormal and normal patterns obtained from history logs
US10733149B2 (en) Template based data reduction for security related information flow data
US20220006828A1 (en) System and user context in enterprise threat detection
US20170185468A1 (en) Creating A Correlation Rule Defining A Relationship Between Event Types
US9590880B2 (en) Dynamic collection analysis and reporting of telemetry data
US20190163553A1 (en) Automated problem diagnosis on logs using anomalous telemetry analysis
US20170178026A1 (en) Log normalization in enterprise threat detection
US20170178025A1 (en) Knowledge base in enterprise threat detection
US20200112475A1 (en) Real-time adaptive infrastructure scenario identification using syntactic grouping at varied similarity
CN104636130A (en) Method and system for generating event trees
US20090307508A1 (en) Optimizing the Efficiency of an Organization's Technology Infrastructure
US20120185584A1 (en) Recording application consumption details
US20200110815A1 (en) Multi contextual clustering
US11501058B2 (en) Event detection based on text streams
EP4254239A1 (en) Event data processing
EP4254198A2 (en) Event data processing
US9659266B2 (en) Enterprise intelligence (‘EI’) management in an EI framework
US9639815B2 (en) Managing processes in an enterprise intelligence (‘EI’) assembly of an EI framework
US20130019246A1 (en) Managing A Collection Of Assemblies In An Enterprise Intelligence ('EI') Framework
US11366712B1 (en) Adaptive log analysis
CN114595118A (en) Data monitoring method and device, storage medium and electronic equipment
US9646278B2 (en) Decomposing a process model in an enterprise intelligence (‘EI’) framework
CN115033639A (en) Method and related device for generating relation graph for data sharing among clusters
Chen et al. System-Level Data Management for Endpoint Advanced Persistent Threat Detection: Issues, Challenges and Trends
CN111177704B (en) Binding identification method, binding identification device, binding identification equipment and binding identification medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination