CN114546780A - Data monitoring method, device, equipment, system and storage medium - Google Patents
Data monitoring method, device, equipment, system and storage medium Download PDFInfo
- Publication number
- CN114546780A CN114546780A CN202210181450.XA CN202210181450A CN114546780A CN 114546780 A CN114546780 A CN 114546780A CN 202210181450 A CN202210181450 A CN 202210181450A CN 114546780 A CN114546780 A CN 114546780A
- Authority
- CN
- China
- Prior art keywords
- data
- monitoring
- analysis
- original
- logic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
Landscapes
- Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Mathematical Physics (AREA)
- Quality & Reliability (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a data monitoring method, a device, equipment, a system and a storage medium, wherein the data monitoring method is applied to a data monitoring platform arranged at a server and comprises the following steps: the method comprises the steps of collecting original monitoring data carrying business scene information from a target application, obtaining analysis logic of the original monitoring data according to the business scene information, carrying out data analysis on the original monitoring data by using the analysis logic, and writing the analysis data into a database. According to the method, data analysis is performed without setting a buried point in the target application, the functional redundancy of the front-end target application is reduced, the invasion and the resource consumption of the target application are avoided, and therefore the running efficiency of the target application is improved.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data monitoring method, apparatus, device, system, and storage medium.
Background
Currently, for monitoring general application real-time indexes or data, a developer sets buried points one by one in application through an SDK (software development kit) to collect data and store the data in a database, then configures abnormal data detection rules, and queries database data at regular time for monitoring.
In the case of large data volume, collecting data consumes more resources, and is highly invasive to code and difficult to maintain. When a database is queried, if complex query logics like aggregation exist, data can be too heavy, response time becomes slow, and even the data is broken down.
Disclosure of Invention
Aiming at the problems in the prior art, the invention aims to provide a data monitoring method, a data monitoring device, data monitoring equipment, a data monitoring system and a storage medium, which overcome the difficulties in the prior art and can improve the running efficiency of application.
The embodiment of the invention provides a data monitoring method, which is applied to a data monitoring platform arranged at a server side and comprises the following steps:
acquiring original monitoring data carrying business scene information from a target application;
acquiring analysis logic of the original monitoring data according to the service scene information, and performing data analysis on the original monitoring data by using the analysis logic;
and writing the analysis data into the database.
In an optional embodiment, obtaining, according to the service scenario information, an analysis logic for the original monitoring data includes:
and under the condition that the created data analysis task is inquired according to the service scene information, extracting the data format information and the corresponding analysis logic of the original monitoring data according to the data analysis task.
In an optional embodiment, obtaining, according to the service scenario information, an analysis logic for the original monitoring data includes:
and under the condition that the created data analysis task is not inquired according to the service scene information, under the condition that a first input of a user is received, responding to the first input of the user, and obtaining data format information and analysis logic corresponding to the service scene information.
In an optional embodiment, obtaining, according to the service scenario information, an analysis logic for the original monitoring data includes:
and acquiring data format information and corresponding analysis logic of the original monitoring data by calling a model contract management platform interface corresponding to the service scene information.
In an alternative embodiment, the data parsing of the raw monitoring data using parsing logic includes:
and under the condition that the second input of the user is received and the target field in the original monitoring data is obtained in response to the second input of the user, carrying out data analysis on the target field by utilizing the analysis logic.
In an alternative embodiment, writing the parsed data to a database includes:
and aggregating the analyzed data, and writing the aggregated data into the database.
In an optional embodiment, the data monitoring method further comprises:
extracting target monitoring data from a database;
and carrying out abnormal data detection on the target monitoring data based on the configured alarm rule, and sending an alarm under the condition that the result shows abnormality.
In an optional embodiment, the detecting abnormal data of the target monitoring data based on the configured alarm rule includes:
and under the condition that the third input of the user is received and the configured alarm rule is obtained in response to the third input of the user, abnormal data detection is carried out on the target monitoring data on the basis of the configured alarm rule.
In an optional embodiment, before performing abnormal data detection on the target monitoring data based on the configured alarm rule, the data monitoring method further includes:
and displaying the configuration template, and automatically generating a monitoring page and configured alarm rules based on the configuration information and the configuration template under the condition that the configuration information is obtained in response to the third input of the user.
An embodiment of the present invention further provides a data monitoring system, where the system includes:
the target application generates original monitoring data of at least one service scene and sends the original monitoring data carrying service scene information;
the data monitoring platform is arranged at the server side and used for receiving original monitoring data carrying business scene information from a target application, acquiring analysis logic of the original monitoring data according to the business scene information, carrying out data analysis on the original monitoring data by using the analysis logic and writing the analysis data into a database.
The embodiment of the present invention further provides a data monitoring apparatus, which is applied to a data monitoring platform disposed at a server, and includes:
the acquisition module is used for acquiring original monitoring data carrying business scene information from the target application;
the data analysis module acquires analysis logic of the original monitoring data according to the service scene information and performs data analysis on the original monitoring data by using the analysis logic;
and the database writing module writes the analysis data into the database.
An embodiment of the present invention further provides an electronic device, including:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the above-described data monitoring method via execution of executable instructions.
Embodiments of the present invention also provide a computer-readable storage medium for storing a program, which when executed implements the steps of the above-mentioned data monitoring method.
The invention aims to provide a data monitoring method, a data monitoring device, data monitoring equipment, a data monitoring system and a storage medium.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, with reference to the accompanying drawings.
FIG. 1 is a block diagram of a data monitoring system in accordance with one embodiment of the present invention;
FIG. 2 is a block diagram of a data monitoring platform according to one embodiment of the present invention;
FIG. 3 is a flow diagram of a data monitoring method of one embodiment of the present invention;
FIG. 4 is a block diagram of a data monitoring device in accordance with one embodiment of the present invention;
FIG. 5 is a block diagram of a data monitoring device according to yet another embodiment of the present invention;
FIG. 6 is a schematic illustration of the operation of the data monitoring device of the present invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.
The drawings are merely schematic illustrations of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware forwarding modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
Through research on data acquisition schemes of related technologies, it is found that data needs to be acquired by a front-end application and data preprocessing such as necessary data analysis is performed on the data, and a server acquires the preprocessed data from the front-end application and performs abnormal data detection.
Data parsing is a method for converting a string of data into different types of data, and thus is a data format conversion process.
In this case, the redundancy of the front-end application functions, the high intrusiveness of the data acquisition and the corresponding preprocessing codes to the application, the consumption of excessive resources, and the slow response time and even the breakdown of the application are one reason.
The embodiment of the present disclosure sets a data monitoring platform at a server corresponding to a target application, where the data monitoring platform is used for executing a data acquisition method, and the data monitoring platform specifically includes:
the method comprises the steps of collecting original monitoring data and corresponding service scene information from a target application, obtaining analysis logic of the original monitoring data according to the service scene information, carrying out data analysis on the original monitoring data by using the analysis logic, and writing the analysis data into a database.
In the embodiment of the disclosure, the original monitoring data generated by the target application carries the service scene information, so that the server can acquire corresponding analysis logic according to the service scene information to perform data analysis on the original monitoring data from the target application, and a data analysis task in a data acquisition process can be realized at the server, so that the functional redundancy of the front-end application is reduced, the intrusion and the resource consumption of the target application are avoided, and the operating efficiency of the target application is improved.
In the disclosed embodiment, the parsed data written into the database using the above method can be used for detection of abnormal data.
Fig. 1 illustrates a structure of a data monitoring system provided in an embodiment of the present disclosure, and as illustrated in fig. 1, the data monitoring system includes: the system comprises a target application 11 and a data monitoring platform 12 arranged at a server, wherein:
the target application 11 is configured to generate original monitoring data of at least one service scenario and send the original monitoring data carrying service scenario information;
the data monitoring platform 12 is configured to receive original monitoring data carrying service scene information from the target application 11, obtain an analysis logic for the original monitoring data according to the service scene information, perform data analysis on the original monitoring data by using the analysis logic, and write the analysis data into a database.
According to the data monitoring platform, the server side collects original monitoring data carrying corresponding service scene information from the target application and executes data analysis, so that data analysis is executed without setting a buried point in the target application, functional redundancy of the front-end target application is reduced, intrusion and resource consumption of the target application are avoided, and running efficiency of the target application is improved.
Referring to fig. 2, in an alternative embodiment of the present disclosure, the data monitoring platform 20 specifically includes the following modules:
the data acquisition module 21 is used for acquiring original monitoring data carrying business scene information from a target application;
and the data processing module 22 acquires analysis logic of the original monitoring data according to the service scene information, performs data analysis on the original monitoring data by using the analysis logic, and writes the analysis data into the database.
In an alternative embodiment, the data processing module 22 may also perform data aggregation on the parsed data, and write the aggregated data into the database.
In an alternative embodiment, the data monitoring platform 20 may further include:
the abnormal data detection module 23 queries the target monitoring data from the database, performs abnormal data detection on the target monitoring data, and sends an alarm prompt when the detection result shows abnormality.
By providing the abnormal data detection module 23 at the server side, this can reduce the functional redundancy of the database.
In other alternative embodiments, the abnormal data alarm module may also be disposed in the database.
In the embodiment of the present disclosure, the data acquisition module 21 is further specifically configured to pack the original monitoring data, and send the packed data to the data processing module 22;
the data processing module 22 is specifically configured to decompress the packed data to obtain original monitoring data, and perform data analysis on the original monitoring data.
In this embodiment, the data acquisition module 21 is configured to be compatible with any data format, and supports data packaging and transmission. Specifically, the original monitoring data can be packaged into an object according to a format defined in advance, the object is directly converted into a binary data form and is sent to the message queue server, and information such as scenes, message numbers and the like is sent at the same time.
In practical applications, for example, a user packages all original monitoring data input into a model at a time into a data object in a Protocol (google Protocol) format, the data acquisition module acquires the object and converts the object into a data format of a binary array, and simultaneously sends scene and message number information to a message queue server, and the message is temporarily stored in the message queue server.
The data processing module 22 obtains information in the message queue in real time, obtains all scene information, and performs data processing operations such as corresponding data analysis.
In practical application, after the data format file is obtained, one data in the message queue can be converted into one row of data after being analyzed, and the row of data is written into a database.
Fig. 3 is a flowchart of a data monitoring method provided in an embodiment of the present disclosure, where the method is applied to a data monitoring platform disposed at a server. As shown in fig. 3, the data monitoring method includes the following steps:
step 310: acquiring original monitoring data carrying business scene information from a target application;
step 320: acquiring analysis logic of the original monitoring data according to the service scene information, and performing data analysis on the original monitoring data by using the analysis logic;
step 330: and writing the analysis data into the database.
The monitoring data come from the target application and can reflect the user behavior, so that the monitoring data can be used for detecting the abnormity. And the original monitoring data is generated by the target application and is directly sent to the data monitoring platform under the condition of not performing data analysis and other processing.
In practice, at least one service scenario may be integrated into the target application, and each service scenario may generate raw monitoring data related to the current scenario during operation.
Based on the service difference between service scenes, the data format of the original monitoring data in each service scene may be different, and the data format (data format) refers to a data formatting format, which may be in the form of a numerical value, a character, or a binary number, and may be described by a data type and a data length. The parsing logic specifically refers to parsing the original monitoring data into a data format supported by the database.
The analysis logic of the original monitoring data of each service scene is different due to the data format difference of the original monitoring data, or the corresponding service scene has the corresponding analysis logic.
And a data monitoring platform of the server acquires original monitoring data under a corresponding service scene from the target application, and performs data processing operations such as data analysis on the original monitoring data.
In the embodiment of the present disclosure, obtaining an analysis logic of original monitoring data according to service scenario information specifically includes the following steps:
and under the condition that the created data analysis task is inquired according to the service scene information, extracting the data format information and the corresponding analysis logic of the original monitoring data according to the data analysis task.
In this case, the analysis logic can perform format conversion on the original monitoring data corresponding to the data format information.
In this embodiment, in the data monitoring platform, the parsing logic and the data parsing task based on the parsing logic are set for the corresponding service scenario in advance, so that the data format and the corresponding parsing logic of the original monitoring data can be obtained in real time according to the data parsing task.
In an optional embodiment of the present disclosure, obtaining an analysis logic of original monitoring data according to service scenario information specifically includes the following steps:
and under the condition that the created data analysis task is not inquired according to the service scene information, under the condition that a first input of a user is received, responding to the first input of the user, and obtaining data format information and analysis logic corresponding to the service scene information.
In this case, a data parsing task may be created according to the data format information and parsing logic corresponding to the service scenario information. Therefore, corresponding data format information and analysis logic can be extracted according to the data analysis task aiming at the monitoring data in the same service scene in the follow-up process.
The data monitoring platform of the embodiment provides a user interface, and receives a first input of a user through the user interface. Wherein the first input may be a text input or other form of input.
In an optional embodiment of the present disclosure, the data format information and the corresponding parsing logic of the original monitoring data may also be automatically obtained by calling a model contract management platform interface corresponding to the service scenario information.
In this embodiment, the original monitoring data from the target application is input to the service processing model for outputting the service response result, and is input to the data monitoring platform set at the server for data analysis and the like. Because the original monitoring data of the corresponding service scene has the corresponding service processing model, the data format information of the original monitoring data input into the corresponding service processing model can be obtained by calling the model contract management platform interface.
In the disclosed embodiment, the parsed data supports the database format, and thus can be normally written into the data table. In this case, the data analysis of the original monitoring data by the analysis logic includes the following steps:
and acquiring a data format of the original monitoring data, and performing data format conversion on the original monitoring data according to the analysis logic, wherein the data after the data format conversion is used as analysis data and written into a corresponding data table in a database.
In the embodiment of the present disclosure, the data analysis of the original monitoring data by using the analysis logic includes:
and under the condition that the second input of the user is received and the target field in the original monitoring data is obtained in response to the second input of the user, carrying out data analysis on the target field by utilizing the analysis logic.
In this embodiment, the data monitoring platform may display a user configuration interface, display the original monitoring data through the user configuration interface, and receive a selection of a user for a target field in the original monitoring data. The user can select to analyze or not analyze the specific field, and data analysis is realized according to the user requirements, so that good user experience is brought.
In the embodiment of the present disclosure, the data analysis of the original monitoring data by using the analysis logic includes:
and under the condition that the input of a user is received and the format of the target data table is obtained in response to the input of the user, analyzing the original monitoring data based on the format of the target data table by using an analyzing logic, and then, analyzing the data to accord with the format of the target data table. The target data table is a data table to be written into the database.
In this embodiment, the parsing logic performs format conversion on the original monitoring data to obtain a data format supported by the database. Further, a user can select a data table through the data monitoring platform, so that the user customization of the data table is realized.
In this embodiment, the user refers to a developer on the server side or a corresponding user.
In an optional embodiment of the present disclosure, writing the analysis data into the database specifically includes:
and aggregating the analyzed data, and writing the aggregated data into the database.
Data Aggregation (Data Aggregation) refers to merging Data from different Data sources, and in particular also to a Data transformation process that is capable of generating scalar values from an array. In this embodiment, the data aggregation may refer to merging data from different service scenarios, and may also refer to a data extraction and combination process for obtaining a value from parsing data.
In the exemplary embodiment of the present disclosure, the data monitoring platform provides an aggregation field configuration service, can obtain a field to be aggregated in the parsed data in response to an input of a user, and performs data aggregation on the field to be aggregated.
In addition, an aggregation logic configuration interface is further provided, aggregation logic configuration information is received through the aggregation logic configuration interface, aggregation logic is obtained through the aggregation logic configuration information, and data aggregation is performed on the analysis data through the aggregation logic.
The embodiment of the disclosure can perform pre-polymerization before the analysis data is stored in the database, so as to reduce the load of the database, thereby improving the query response speed of the database.
In the embodiment of the present disclosure, each service scenario may generate a real-time computing task, and after the task acquires original monitoring data, it first determines whether the data is data of a current service scenario, and if not, discards the data; if yes, the original monitoring data is analyzed and formatted. If the analyzed data needs to be aggregated, the data aggregation is carried out after the formatting; and finally, writing the aggregated data into the distributed database.
In practical application, after the data format file is obtained, one data in the message queue can be converted into one line of data after being analyzed, and the line of data is written into a database. Some data may be selected to be aggregated after being analyzed, for example, the data may be analyzed as an array type data, and the average value, length, etc. of the array may be calculated through aggregation.
In the embodiment of the present disclosure, the method may further include:
extracting target monitoring data from a database;
and carrying out abnormal data detection on the target monitoring data based on the configured alarm rule, and sending an alarm under the condition that the result shows abnormality.
The data monitoring platform of the embodiment configures an abnormal data detection function, which can reduce consumption of resources related to services in the server.
In the embodiment of the disclosure, under the condition that the third input of the user is received and the configured alarm rule is obtained in response to the third input of the user, the abnormal data detection is performed on the target monitoring data based on the configured alarm rule.
In this embodiment, the data monitoring platform provides an alarm rule configuration function, and a user can configure an alarm rule through the data monitoring platform.
In an optional embodiment of the present disclosure, the data monitoring platform displays the configuration template, and automatically generates a monitoring page and configured alarm rules based on the configuration information and the configuration template in a case where the configuration information is obtained in response to a third input of the user.
Alternatively, the user may select a specific field to generate the monitoring page, so that the specific field to be detected may be displayed in the monitoring page.
In an optional embodiment of the present disclosure, a user may update a monitoring index in the alarm rule through the monitoring page, such as modifying or adding the monitoring index, so that the data monitoring platform may update the alarm rule based on the updated monitoring index. For example, the monitoring metrics may be updated by modifying the Structured Query Language (SQL).
The monitoring indexes correspond to corresponding magnitude values in the data fields, and the monitoring indexes are used as the basis for abnormal data detection.
In this case, whether an alarm is triggered may be calculated based on configured alarm rules and thresholds.
In an alternative embodiment, the data monitoring platform can query the data in the database regularly and extract the target monitoring data.
In this case, the data monitoring platform also provides a setting interface of query frequency, alarm rule, alarm threshold value, etc., so that the user can modify the query frequency, the alarm rule or the alarm threshold value.
In the embodiment of the present disclosure, the alarm mode may be a telephone, a short message, or an email.
Wherein, different alarm rules can configure different alarm levels and notification modes.
In order to reduce false alarm, the user can select to trigger the alarm threshold value for multiple times and then send the alarm.
In an exemplary embodiment of the present disclosure, the data monitoring platform may receive an input from a user, obtain a monitoring page to be presented in response to the input from the user, and jump to the page.
In practical application, a user can create a big data processing task through a configuration interface of a data monitoring platform, wherein the configuration comprises service scene name selection, such as ClickHouse distributed database configuration (database name and data table name), Kafka message queue configuration (message subject and message type), UDF function configuration (function name and function type) in a Flink real-time processing task, and the big data processing task can be generated after click determination.
After the data is written into the distributed database, the user can generate a monitoring page on the data monitoring platform.
In practical application, a user can generate monitoring of certain characteristics in batch, configure monitoring importance levels and monitoring indexes, query frequency and the like.
And a user-defined monitoring item can be manually added, for example, the following SQL can be used for calculating the average value of each minute of a certain characteristic in a certain period of time, regularly inquiring the SQL statement, and judging whether to alarm or not according to a configured alarm rule (for example, the average value of the last five minutes is greater than 1).
SELECT
toUInt32(toStartOfMinute(eventtime))*1000as t,
avg($featureName)
FROM$table
WHERE eventtime>＝toDateTime(1640156160)AND eventtime<toDateTime(1640166660)
GROUP BY t
ORDER BY t
The data monitoring platform can be provided with a data query module, an index calculation module and an alarm judgment module, the data query module is a query component connected with the database, and target monitoring data can be returned when SQL and the database name are transmitted. And the index calculation module calculates indexes such as a mean value, a 1-hour change rate, a 24-hour change rate and the like according to the query result.
The alarm judging module judges whether to trigger an alarm (if the average value is continuously lower than 0.01 for 10 minutes) according to the calculation index, if the input result of the alarm judging module is the trigger alarm, the result sending module sends the alarm according to the preset configuration, and the notification mode can be mail, short message, telephone and the like.
FIG. 4 is a block diagram of an embodiment of a data monitoring device of the present invention. The data monitoring apparatus 400 of the present invention is applied to a data monitoring platform disposed at a server, as shown in fig. 4, including but not limited to:
an acquisition module 410, which acquires original monitoring data carrying service scene information from a target application;
the data analysis module 420 acquires analysis logic of the original monitoring data according to the service scene information, and performs data analysis on the original monitoring data by using the analysis logic;
the database write module 430 writes the parsed data to the database.
The implementation principle of the above modules is described in the data monitoring method, and is not described herein again.
The data monitoring device provided by the embodiment of the invention is arranged at the server side, and the original monitoring data generated by the target application carries the service scene information, so that the server side can acquire corresponding analysis logic according to the service scene information to analyze the data of the original monitoring data from the target application, and the data analysis task in the data acquisition process can be realized at the server side, thereby reducing the functional redundancy of the front-end application, avoiding the invasion and resource consumption of the target application, and improving the operating efficiency of the target application.
Optionally, the data parsing module 420 is specifically configured to:
and under the condition that the created data analysis task is inquired according to the service scene information, extracting the data format information and the corresponding analysis logic of the original monitoring data according to the data analysis task.
Optionally, the data parsing module 420 is specifically configured to:
and under the condition that the created data analysis task is not inquired according to the service scene information, under the condition that a first input of a user is received, responding to the first input of the user, and obtaining data format information and analysis logic corresponding to the service scene information.
Optionally, the data parsing module 420 is specifically configured to:
and acquiring data format information and corresponding analysis logic of the original monitoring data by calling a model contract management platform interface corresponding to the service scene information.
Optionally, the data parsing module 420 is specifically configured to:
and under the condition that the second input of the user is received and the target field in the original monitoring data is obtained in response to the second input of the user, carrying out data analysis on the target field by utilizing the analysis logic.
Optionally, the database writing module 430 is specifically configured to:
and aggregating the analyzed data, and writing the aggregated data into the database.
Optionally, compared to fig. 4, the data monitoring apparatus 500 shown in fig. 5 further includes:
an abnormal data detection module 510 that extracts target monitoring data from the database;
and carrying out abnormal data detection on the target monitoring data based on the configured alarm rule, and sending an alarm under the condition that the result shows abnormality.
Optionally, the abnormal data detecting module 510 is specifically configured to:
and under the condition that a third input of the user is received and the configured alarm rule is obtained in response to the third input of the user, abnormal data detection is carried out on the target monitoring data based on the configured alarm rule.
Optionally, the abnormal data detecting module 510 is further specifically configured to:
before abnormal data detection is carried out on the target monitoring data based on the configured alarm rule, a configuration template is displayed, and under the condition that configuration information is obtained in response to third input of a user, a monitoring page and the configured alarm rule are automatically generated based on the configuration information and the configuration template.
The embodiment of the invention also provides electronic equipment which comprises a processor. A memory having stored therein executable instructions of the processor. Wherein the processor is configured to perform the steps of the data monitoring method via execution of the executable instructions.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" platform.
Fig. 6 is a schematic structural diagram of the data monitoring apparatus of the present invention. An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 6. The electronic device 600 shown in fig. 6 is only an example and should not bring any limitation to the function and the scope of use of the embodiments of the present invention.
As shown in fig. 6, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different platform components (including the memory unit 620 and the processing unit 610), a display unit 640, etc.
Where the storage unit stores program code which may be executed by the processing unit 610 such that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention as described in the data monitoring method section of this specification. For example, processing unit 610 may perform the steps as shown in fig. 3.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)621 and/or a cache memory unit 622, and may further include a read only memory unit (ROM) 623.
The storage unit 620 may also include a program/utility 624 having a set (at least one) of program modules 625, such program modules 625 including, but not limited to: a processing system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 670 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650.
Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, to name a few.
The embodiment of the invention also provides a computer readable storage medium for storing the program, and the steps of the data monitoring method are realized when the program is executed. In some possible embodiments, the various aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the above-mentioned data monitoring method section of the present description, when the program product is run on the terminal device.
According to the program product for realizing the method, the portable compact disc read only memory (CD-ROM) can be adopted, the program code is included, and the program product can be operated on terminal equipment, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out processes of the present invention may be written in any combination of one or more programming languages, including object oriented programming languages such as Java, C + + or the like and conventional procedural programming languages, such as "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, numerous simple deductions or substitutions may be made without departing from the spirit of the invention, which shall be deemed to belong to the scope of the invention.
Claims (13)
1. A data monitoring method is applied to a data monitoring platform arranged at a server side, and comprises the following steps:
acquiring original monitoring data carrying business scene information from a target application;
acquiring analysis logic of the original monitoring data according to the service scene information, and performing data analysis on the original monitoring data by using the analysis logic;
and writing the analysis data into the database.
2. The data monitoring method according to claim 1, wherein obtaining parsing logic for the original monitoring data according to the service scenario information comprises:
and under the condition that the created data analysis task is inquired according to the service scene information, extracting the data format information of the original monitoring data and the corresponding analysis logic according to the data analysis task.
3. The data monitoring method according to claim 1, wherein obtaining parsing logic for the original monitoring data according to the service scenario information comprises:
and under the condition that the created data analysis task is not inquired according to the service scene information, under the condition that a first input of a user is received, responding to the first input of the user, and obtaining data format information and analysis logic corresponding to the service scene information.
4. The data monitoring method according to claim 1, wherein obtaining parsing logic for the original monitoring data according to the service scenario information comprises:
and acquiring data format information and corresponding analysis logic of the original monitoring data by calling a model contract management platform interface corresponding to the service scene information.
5. The data monitoring method of claim 1, wherein performing data parsing on the raw monitoring data using the parsing logic comprises:
and under the condition that a second input of the user is received and a target field in the original monitoring data is obtained in response to the second input of the user, carrying out data analysis on the target field by using the analysis logic.
6. The data monitoring method of claim 1, wherein writing parsed data to a database comprises:
and aggregating the analyzed data, and writing the aggregated data into the database.
7. The data monitoring method of claim 1, further comprising:
extracting target monitoring data from a database;
and carrying out abnormal data detection on the target monitoring data based on a configured alarm rule, and sending an alarm under the condition that the result shows abnormal.
8. The data monitoring method according to claim 7, wherein performing abnormal data detection on the target monitoring data based on configured alarm rules comprises:
and under the condition that a third input of a user is received and the configured alarm rule is obtained in response to the third input of the user, carrying out abnormal data detection on the target monitoring data based on the configured alarm rule.
9. The data monitoring method of claim 8, wherein before performing abnormal data detection on the target monitoring data based on the configured alarm rule, the data monitoring method further comprises:
and displaying a configuration template, and automatically generating a monitoring page and the configured alarm rule based on the configuration information and the configuration template under the condition that the configuration information is obtained in response to the third input of the user.
10. A data monitoring system, comprising:
the system comprises a target application and a server, wherein the target application generates original monitoring data of at least one service scene and sends the original monitoring data carrying service scene information;
the data monitoring platform is arranged at a server side, receives original monitoring data carrying business scene information from the target application, acquires analysis logic of the original monitoring data according to the business scene information, performs data analysis on the original monitoring data by using the analysis logic, and writes the analysis data into a database.
11. The utility model provides a data monitoring device which characterized in that, is applied to and sets up in the data monitoring platform of server, includes:
the acquisition module is used for acquiring original monitoring data carrying business scene information from the target application;
the data analysis module acquires analysis logic of the original monitoring data according to the service scene information and performs data analysis on the original monitoring data by using the analysis logic;
and the database writing module writes the analysis data into the database.
12. An electronic device, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the data monitoring method of any one of claims 1 to 9 via execution of the executable instructions.
13. A computer-readable storage medium storing a program which, when executed by a processor, performs the steps of the data monitoring method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210181450.XA CN114546780A (en) | 2022-02-25 | 2022-02-25 | Data monitoring method, device, equipment, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210181450.XA CN114546780A (en) | 2022-02-25 | 2022-02-25 | Data monitoring method, device, equipment, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114546780A true CN114546780A (en) | 2022-05-27 |
Family
ID=81679265
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210181450.XA Pending CN114546780A (en) | 2022-02-25 | 2022-02-25 | Data monitoring method, device, equipment, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114546780A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115118620A (en) * | 2022-06-24 | 2022-09-27 | 银清科技有限公司 | Configurable monitoring flow method and system |
-
2022
- 2022-02-25 CN CN202210181450.XA patent/CN114546780A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115118620A (en) * | 2022-06-24 | 2022-09-27 | 银清科技有限公司 | Configurable monitoring flow method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| CN111190888A (en) | Method and device for managing graph database cluster | |
| CN112636957B (en) | Early warning method and device based on log, server and storage medium | |
| CN111309550A (en) | Data acquisition method, system, equipment and storage medium of application program | |
| CN112084224B (en) | Data management method, system, equipment and medium | |
| CN112000806A (en) | Abnormal log monitoring and analyzing method, system, equipment and storage medium | |
| CN112615742A (en) | Method, device, equipment and storage medium for early warning | |
| CN115422003A (en) | Data quality monitoring method and device, electronic equipment and storage medium | |
| CN114546780A (en) | Data monitoring method, device, equipment, system and storage medium | |
| US9659266B2 (en) | Enterprise intelligence (‘EI’) management in an EI framework | |
| CN116955856A (en) | Information display method, device, electronic equipment and storage medium | |
| CN114697247B (en) | Fault detection method, device, equipment and storage medium of streaming media system | |
| CN110677271A (en) | Big data alarm method, device, equipment and storage medium based on ELK | |
| CN112948341B (en) | Method and apparatus for identifying anomalous network device logs | |
| CN113590447B (en) | Buried point processing method and device | |
| CN115904369A (en) | Method and system for efficient aggregation and correlation analysis of network security source data | |
| CN114756301A (en) | Log processing method, device and system | |
| CN115454443A (en) | Multi-system data interaction method and device and electronic equipment | |
| CN115408236A (en) | Log data auditing system, method, equipment and medium | |
| CN113342619A (en) | Log monitoring method and system, electronic device and readable medium | |
| CN114995719A (en) | List rendering method, device, equipment and storage medium | |
| US20170337644A1 (en) | Data driven invocation of realtime wind market forecasting analytics | |
| CN108874625B (en) | Information processing method and device, electronic equipment and storage medium | |
| CN112712443A (en) | Event analysis method and analysis device for converter station | |
| CN112749204A (en) | Method and device for reading data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |