CN113760640A - Monitoring log processing method, device, equipment and storage medium - Google Patents
Monitoring log processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN113760640A CN113760640A CN202011271126.4A CN202011271126A CN113760640A CN 113760640 A CN113760640 A CN 113760640A CN 202011271126 A CN202011271126 A CN 202011271126A CN 113760640 A CN113760640 A CN 113760640A
- Authority
- CN
- China
- Prior art keywords
- monitoring
- calling
- cache
- time period
- piece
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 457
- 238000003672 processing method Methods 0.000 title claims abstract description 36
- 238000000034 method Methods 0.000 claims abstract description 134
- 238000004458 analytical method Methods 0.000 claims abstract description 73
- 238000013500 data storage Methods 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims description 94
- 238000012806 monitoring device Methods 0.000 claims description 69
- 230000008569 process Effects 0.000 claims description 25
- 238000004590 computer program Methods 0.000 claims description 11
- 238000010586 diagram Methods 0.000 description 25
- 238000013461 design Methods 0.000 description 12
- 238000004891 communication Methods 0.000 description 7
- 230000003139 buffering effect Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 230000003993 interaction Effects 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 101100116390 Schizosaccharomyces pombe (strain 972 / ATCC 24843) ded1 gene Proteins 0.000 description 1
- 101100311460 Schizosaccharomyces pombe (strain 972 / ATCC 24843) sum2 gene Proteins 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3058—Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Computer Hardware Design (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the application provides a monitoring log processing method, a device, equipment and a storage medium, a plurality of pieces of calling data generated at each monitoring point when a target application calls a designated method in each caching time period are obtained, at least one piece of calling information corresponding to the plurality of pieces of calling data is sequentially stored into a corresponding caching queue according to a monitoring point identifier and a caching time period identifier carried by each piece of calling data, all pieces of calling information in each caching queue are processed at intervals of time corresponding to the caching time period, a plurality of monitoring logs aiming at the target application are generated and transmitted to the monitoring equipment, the monitoring equipment respectively caches the plurality of monitoring logs according to a data storage template preset by each monitoring point, the plurality of cached monitoring logs are processed according to preset analysis frequency, and the performance index of the target application is determined, the method is not influenced by the QPS of the target application, and the performance analysis efficiency and the analysis accuracy of the application are improved.
Description
Technical Field
The embodiment of the application relates to the technical field of computers, in particular to a monitoring log processing method, a monitoring log processing device, monitoring log processing equipment and a storage medium.
Background
With the continuous development of distributed application and cloud computing technologies, the logic structure of a business system becomes more and more complex, and applications have evolved into a series of services running on different platforms. The performance of the application is a key index for reflecting the quality of service provided by the application program to the client, and the use experience of the user is directly influenced by the level of the performance of the application.
In the prior art, in order to monitor the running performance of an application, the running performance is generally determined based on the performance condition of a specified method executed by the application. Specifically, each time the application calls the designated method, a monitoring log is generated and synchronously written into a disk log file, a log acquisition module periodically acquires incremental log data from the disk log file and sends the incremental log data to the monitoring equipment, and correspondingly, the monitoring equipment caches and analyzes the received incremental log data to determine the performance data of the designated method, so that the running performance of the application is determined.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art: in practical application, if the query rate per second (QPS) of a certain application to be monitored is relatively high, a large number of times of calling the specified method at this time will generate a large number of monitoring logs in a short time, which not only increases the refresh frequency of disk log files and increases the acquisition pressure of a log acquisition module, but also seriously affects the normal operation of the application or causes data loss, and occupies a large amount of cache space of the monitoring device, increases the processing pressure of the monitoring device, and causes a reduction in performance analysis efficiency and accuracy. In general, the existing performance method has the problems of low performance analysis efficiency and accuracy.
Disclosure of Invention
The embodiment of the application provides a monitoring log processing method, a monitoring log processing device, monitoring log processing equipment and a monitoring log storage medium, and aims to solve the problems of low performance analysis efficiency and accuracy in the existing performance method.
In a first aspect, an embodiment of the present application provides a monitoring log processing method, including:
acquiring a plurality of pieces of calling data generated at each monitoring point when a target application calls a specified method in each caching time period, wherein each piece of calling data carries a monitoring point identifier and a caching time period identifier;
according to the monitoring point identification and the cache time period identification carried by each piece of calling data, at least one piece of calling information corresponding to the calling data is sequentially stored into a corresponding cache queue, and each piece of calling information comprises: calling execution time and calling execution times of the calling execution time, wherein each cache queue is named by a monitoring point identifier and a cache time period identifier;
processing all calling information in each cache queue at intervals of a time length corresponding to the cache time period to generate a plurality of monitoring logs for the target application;
and sending the monitoring logs to a monitoring device.
In a possible design of the first aspect, the sequentially storing at least one piece of call information corresponding to the plurality of pieces of call data in a corresponding cache queue according to a monitor point identifier and a cache time period identifier carried in each piece of call data includes:
dividing the obtained calling data into at least one data set according to a monitoring point identifier and a cache time period identifier carried by each piece of calling data, wherein each data set comprises at least one piece of calling data with the same monitoring point identifier and the same cache time period identifier;
for each data set, processing all the calling data in the data set according to the calling execution time of each calling data, and determining at least one piece of calling information;
and sequentially storing each piece of calling information into a cache queue corresponding to the data set.
Optionally, processing all the call information in each cache queue every time interval corresponding to the cache time period to generate a plurality of monitoring logs for the target application, where the processing includes:
and updating the cache time period identification and the monitoring point identification of each cache queue, at least one calling execution time in the cache queue and the calling execution times of each calling execution time into a preset log template every other time length corresponding to the cache time period, and generating a plurality of monitoring logs for the target application.
In another possible design of the first aspect, after processing all the call information in each cache queue at a time corresponding to the interval of the cache time period and generating multiple monitoring logs for the target application, the method further includes:
storing a plurality of monitoring logs of the target application into a disk log file;
acquiring an increment monitoring log in the disk log file according to a preset acquisition period;
the sending the plurality of monitoring logs to the monitoring device includes:
and sending the incremental monitoring log to a monitoring device.
In a second aspect, an embodiment of the present application provides a monitoring log processing method, including:
receiving a plurality of monitoring logs of a target application from an application server;
caching the multiple monitoring logs respectively according to a data storage template preset by each monitoring point;
and processing the plurality of cached monitoring logs according to a preset analysis frequency, and determining the performance index of the target application corresponding to the application server.
In a possible design of the second aspect, the caching the multiple monitoring logs according to a data storage template preset for each monitoring point respectively includes:
analyzing each monitoring log to determine a monitoring point identifier and a cache time period identifier corresponding to each monitoring log;
determining at least one piece of calling information corresponding to each monitoring log according to the monitoring point identifier and the cache time period identifier corresponding to each monitoring log, wherein each piece of calling information comprises: a call execution time and a number of call executions of said call execution time;
and caching at least one piece of calling information corresponding to each monitoring log based on the data storage template corresponding to each monitoring log.
In a third aspect, an embodiment of the present application provides a monitoring log processing method, including:
acquiring a plurality of pieces of calling data generated at each monitoring point when a target application calls a specified method in each caching time period, wherein each piece of calling data carries a monitoring point identifier and a caching time period identifier;
according to the monitoring point identification and the cache time period identification carried by each piece of calling data, at least one piece of calling information corresponding to the calling data is sequentially stored into a corresponding cache queue, and each piece of calling information comprises: calling execution time and calling execution times of the calling execution time, wherein each cache queue is named by a monitoring point identifier and a cache time period identifier;
processing all calling information in each cache queue at intervals of a time length corresponding to the cache time period to generate a plurality of monitoring logs for the target application;
and processing the plurality of monitoring logs according to a preset analysis frequency to determine the performance index of the target application.
In a fourth aspect, an embodiment of the present application provides a monitoring log processing apparatus, including:
the acquisition module is used for acquiring a plurality of pieces of calling data generated at each monitoring point when the target application calls the specified method in each cache time period, and each piece of calling data carries a monitoring point identifier and a cache time period identifier;
the cache module is used for sequentially storing at least one piece of calling information corresponding to the calling data into a corresponding cache queue according to a monitoring point identifier and a cache time period identifier carried by each piece of calling data, and each piece of calling information comprises: calling execution time and calling execution times of the calling execution time, wherein each cache queue is named by a monitoring point identifier and a cache time period identifier;
the processing module is used for processing all calling information in each cache queue at intervals of time corresponding to the cache time period to generate a plurality of monitoring logs for the target application;
and the sending module is used for sending the monitoring logs to the monitoring equipment.
In a possible design of the fourth aspect, the cache module is specifically configured to:
dividing the obtained calling data into at least one data set according to a monitoring point identifier and a cache time period identifier carried by each piece of calling data, wherein each data set comprises at least one piece of calling data with the same monitoring point identifier and the same cache time period identifier;
for each data set, processing all the calling data in the data set according to the calling execution time of each calling data, and determining at least one piece of calling information;
and sequentially storing each piece of calling information into a cache queue corresponding to the data set.
Optionally, the processing module is specifically configured to update, every time duration corresponding to the cache time period, the cache time period identifier and the monitoring point identifier of each cache queue, at least one call execution time in the cache queues, and the call execution times of each call execution time to a preset log template, so as to generate multiple monitoring logs for the target application.
In another possible design of the fourth aspect, the cache module is further configured to process all the call information in each cache queue at a time interval corresponding to the cache time period by the processing module, generate multiple monitoring logs for the target application, and store the multiple monitoring logs of the target application in a disk log file;
the acquisition module is further used for acquiring the incremental monitoring logs in the disk log file according to a preset acquisition period;
the sending module is specifically configured to send the incremental monitoring log to a monitoring device.
In a fifth aspect, an embodiment of the present application provides a monitoring log processing apparatus, including:
the receiving module is used for receiving a plurality of monitoring logs of the target application from the application server;
the cache module is used for caching the plurality of monitoring logs respectively according to a data storage template preset by each monitoring point;
and the processing module is used for processing the plurality of cached monitoring logs according to a preset analysis frequency and determining the performance index of the target application corresponding to the application server.
In a possible design of the fifth aspect, the cache module is specifically configured to:
analyzing each monitoring log to determine a monitoring point identifier and a cache time period identifier corresponding to each monitoring log;
determining at least one piece of calling information corresponding to each monitoring log according to the monitoring point identifier and the cache time period identifier corresponding to each monitoring log, wherein each piece of calling information comprises: a call execution time and a number of call executions of said call execution time;
and caching at least one piece of calling information corresponding to each monitoring log based on the data storage template corresponding to each monitoring log.
In a sixth aspect, an embodiment of the present application provides a monitoring log processing apparatus, including:
the acquisition module is used for acquiring a plurality of pieces of calling data generated at each monitoring point when the target application calls the designated device in each cache time period, and each piece of calling data carries a monitoring point identifier and a cache time period identifier;
the cache module is used for sequentially storing at least one piece of calling information corresponding to the calling data into a corresponding cache queue according to a monitoring point identifier and a cache time period identifier carried by each piece of calling data, and each piece of calling information comprises: calling execution time and calling execution times of the calling execution time, wherein each cache queue is named by a monitoring point identifier and a cache time period identifier;
and the processing module is used for processing all calling information in each cache queue at intervals of time corresponding to the cache time period, generating a plurality of monitoring logs for the target application, processing the plurality of monitoring logs according to a preset analysis frequency and determining the performance index of the target application.
In a seventh aspect, an embodiment of the present application provides an application server, including a processor, a memory, a transceiver, and a computer program stored on the memory and executable on the processor, where the processor executes the computer program to implement the method according to the first aspect and possible designs.
In an eighth aspect, embodiments of the present application provide a monitoring device, including a processor, a memory, a transceiver, and a computer program stored on the memory and executable on the processor, where the processor executes the program to implement the method according to the second aspect.
In a ninth aspect, embodiments of the present application provide an application server, including a processor, a memory, and a computer program stored on the memory and executable on the processor, where the processor executes the program to implement the method according to the third aspect.
In a tenth aspect, embodiments of the present application provide a computer-readable storage medium, in which computer-executable instructions are stored, and when executed by a processor, the computer-executable instructions are configured to implement the method according to the first aspect and various possible designs; or
The computer executable instructions when executed by a processor are for implementing the method as described in the second aspect above; or
The computer executable instructions, when executed by a processor, are for implementing the method as described in the third aspect above.
The monitoring log processing method, the device, the equipment and the storage medium provided by the embodiment of the application, by acquiring a plurality of pieces of calling data generated at each monitoring point when a target application calls a specified method in each caching time period, sequentially storing at least one piece of calling information corresponding to the plurality of pieces of calling data into a corresponding caching queue according to a monitoring point identifier and a caching time period identifier carried by each piece of calling data, processing all pieces of calling information in each caching queue at intervals of time corresponding to the caching time period to generate a plurality of monitoring logs for the target application and transmitting the monitoring logs to the monitoring equipment, so that the monitoring equipment can respectively cache the plurality of monitoring logs according to a data storage template preset by each monitoring point, process the plurality of cached monitoring logs according to preset analysis frequency and determine performance indexes of the target application corresponding to an application server, the method is not influenced by the QPS of the target application, and the performance analysis efficiency and the analysis accuracy of the application are improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.
Fig. 1 is a schematic diagram of an application architecture of a monitoring log processing method according to an embodiment of the present application;
fig. 2 is a schematic diagram of another application architecture of the monitoring log processing method according to the embodiment of the present application;
fig. 3 is a schematic flowchart of a monitoring log processing method according to a first embodiment of the present disclosure;
fig. 4 is a schematic flowchart of a second embodiment of a monitoring log processing method provided in the present application;
fig. 5 is an interaction diagram of a third embodiment of a monitoring log processing method provided by the present application;
fig. 6 is a schematic flowchart of a fourth monitoring log processing method according to the present application;
fig. 7 is a schematic flowchart of a fifth embodiment of a monitoring log processing method provided in the present application;
fig. 8 is a schematic structural diagram of a first monitoring log processing apparatus according to an embodiment of the present disclosure;
fig. 9 is a schematic structural diagram of a second monitoring log processing apparatus according to an embodiment of the present application;
fig. 10 is a schematic structural diagram of a third monitoring log processing apparatus according to an embodiment of the present application;
fig. 11 is a schematic structural diagram of a first embodiment of an application server provided in the present application;
fig. 12 is a schematic structural diagram of an embodiment of a monitoring device provided in an embodiment of the present application;
fig. 13 is a schematic structural diagram of a second application server according to an embodiment of the present application.
With the foregoing drawings in mind, certain embodiments of the disclosure have been shown and described in more detail below. These drawings and written description are not intended to limit the scope of the disclosed concepts in any way, but rather to illustrate the concepts of the disclosure to those skilled in the art by reference to specific embodiments.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.
At this stage, with the rapid development of network technology, various services provided by internet applications, such as e-commerce websites, fund transaction systems, social networking websites, internet application services, are increasingly widely used in the lives of people. The use experience of the user is directly influenced by the level of the application service performance, and how to monitor the running performance of the application service is very critical.
In general, in order to monitor the operation performance of the application service, performance monitoring may be performed on an important method called by the application service, and then the performance of the application service is determined according to the performance of the application method. At present, a relatively general and safe way is to obtain a monitoring log generated at an application server when a target application calls a specified method, and then analyze the monitoring log to determine the performance of the specified method, thereby determining the performance condition of the target application. Optionally, the application server may also report the obtained monitoring log to the monitoring device for analysis, so as to obtain a performance analysis result of the specified method. The embodiment of the present application does not limit the device for performing log analysis.
The following first explains an application architecture of the embodiment of the present application. Fig. 1 is a schematic diagram of an application architecture of a monitoring log processing method according to an embodiment of the present application. As shown in fig. 1, the application architecture may include an application server 11 and a monitoring device 12. Alternatively, in practical applications, the monitoring device 12 may be a monitoring server.
Illustratively, the application server 11 may include: an application module 111, a disk 112, and an acquisition module 113. The application module 111 is mainly used for generating a monitoring log when the target application calls the specified method, and refreshing the monitoring log into a log file of the disk 112, and the acquisition module 113 may read log data from the log file of the disk 112 at regular time and send the log data to the monitoring device 12 for processing.
First, the conventional log processing flow will be described with reference to an application architecture diagram shown in fig. 1, and problems in the conventional log processing flow will be described.
Optionally, in the existing log processing method, the generation and output mode of the monitoring log is the same as the output mode of the normal application service log, and the application module 111 may synchronously calculate the response duration of the specified method when the specified method is called, generate the monitoring log based on a preset monitoring log template, and refresh the monitoring log into the log file of the disk 112. Accordingly, the collection module 113 may collect the log file of the client monitoring disk, and collect incremental log data from the log file of the disk 112 at intervals (e.g., 5 seconds) and send the incremental log data to the monitoring device 12. Therefore, the monitoring device 12 may buffer the received log data according to a preset analysis frequency (e.g., 1 minute), analyze and process the buffered data at regular time, determine the performance index information of the specified method, and further determine the performance of the target application to a certain extent, so as to perform subsequent warning and generate a report.
In practical applications, the monitoring log may contain the call time (time), the monitoring point identification (key), the application name (appName), the host name (hostname), and the call time (elapsedTime, in milliseconds). Correspondingly, the data storage format of the monitoring equipment end is as follows: monitor key- (time-time/frequency of analysis): { hostname: [ elaspedTime1, elaspedTime2, … ] }. Illustratively, the monitoring log generated if the target application calls the specified method in mm minutes and nn seconds at yy month zz dd in xxxx year is represented by the following storage format:
optionally, the monitoring device 12 analyzes the received log data in the following manner. Specifically, the monitoring device may periodically analyze data of a frequency time point from key to last according to the configured monitoring point, for example, the current time is 20200702211730, the preset analysis frequency is 1 minute, and the last analysis frequency time point is the current minute-1, that is, 202007022116.
For example, the data storage format of the monitoring log with the calling time of xxxx yy month zz day dd, where the monitoring device is identified as ap.v2.token.gettoken at the monitoring point, and mm score is as follows:
optionally, after the received log data is buffered, the monitoring device calculates performance indexes of a specified method, such as an average value, a maximum value, a minimum value, TP50, TP90, TP99, and TP999, at the latitude of the whole machine.
Wherein, TP index means: TP-Top Percentile, Top percentage, refers to the time consumed by the method for each call counted over a period of time, and the times are sorted in order from small to large, and the result is taken as: the total times index number (note: percentage) is the value corresponding to the TP index, and then the value corresponding to the sorted time is taken out.
For example, the TP99 value is calculated as follows: all the time-consuming data in the analysis frequency time point are taken out, the time-consuming data is sorted from the small arrival, the 99% +1 time-consuming data is taken as the TP99 value, and the business meaning of the expression is that 99% of the method calls are below the value.
It is worth to be noted that, in practical applications, multiple instances may be deployed in the same application, and each instance may generate a respective monitoring log, so that a specific instance from which each piece of monitoring log data comes may be determined according to host (hostname) field identification data in the monitoring log, and thus, performance of an application method of each hostname instance may be analyzed, and performance of the entire application (i.e., the application) may also be analyzed.
Specifically, in general, the shorter the execution time when a method is called, the higher the performance of the method is considered, but the time for each execution of the same method when the method is called many times is not necessarily consistent, but always stabilizes in a certain interval. Similarly, lower values for these performance indicators indicate higher performance for a given method.
It can be understood that, in practical application, according to the method, not only can the performance of the method be monitored, but also whether an exception occurs when the method is called can be monitored, and if the exception occurs when the method is called, the exception is also reflected in the monitoring log. Regarding the situation of monitoring the abnormality, the embodiment of the present application does not limit the situation, and it can be determined according to actual needs.
In practical applications, if a query rate per second (QPS, which is a measure of how much traffic is processed by a specific query server within a specified time) of an application is relatively high, core processes to be processed by the method for monitoring an application module are relatively large, and under a high concurrency condition, the application module generates a large number of monitoring logs within a short time, which may cause the following problems:
the problem 1 is that the application module generates a large amount of monitoring log data and continuously refreshes the monitoring log data into a log file of a disk, so that the occupancy rates of the disk and a processor of an application server where the application module is located are increased, and the occupancy rates are continuously increased along with the increase of QPS, and the normal operation of the application module is seriously influenced;
problem 2, a large amount of log data is refreshed in a short time by the log file of the magnetic disk, namely, a large incremental log file is generated, the acquisition pressure of an acquisition module is increased, the transmission delay of the log data can be caused because the incremental log file cannot be acquired in time and reported to a monitoring device, and the problem of log data loss caused by too fast segmentation of the log file can be caused in serious cases;
problem 3, the time-consuming value of the process is typically subject to fluctuations within a certain range, for example, within 10-500 milliseconds, when the QPS increases, a large number of repeated time-consuming requests may be generated in a short time (for example, the execution time of the same method call is different but always stabilizes in an interval, for example, the call time stabilization interval of the method a is 10-50 milliseconds, the call is 100 times with at least 50 repeated execution times, the call is 1000 times with at least 950 repeated execution times, the more the call, the more the repeated execution times, and thus the more the generated repeated time-consuming requests), the monitoring device needs to store all the original data after receiving the log data, and thus a large amount of buffer space is occupied, meanwhile, the sequencing of large data volume can also increase the processing pressure of the monitoring equipment, so that a large amount of resources need to be consumed, and the analysis efficiency is influenced.
In conclusion, the existing performance method has the problems of low performance analysis efficiency and accuracy.
Fig. 2 is a schematic diagram of another application architecture of the monitoring log processing method according to the embodiment of the present application. As shown in fig. 2, the monitoring log processing method is applied to an application server 20, and the application server 20 may include: an application module 201, a disk 202, an acquisition module 203 and a monitoring module 204.
As can be seen from the architecture diagrams shown in fig. 2 and fig. 1, in the architecture diagram shown in fig. 2, the monitoring module 204 is integrated in the application server 20, that is, the application server 20 may generate a monitoring log through the application module 201 and refresh the monitoring log into the log file of the disk 202, and the collection module 203 may read log data from the log file of the disk 202 at regular time and transmit the log data to the monitoring module 204 for processing.
The specific implementation of the application module 201, the disk 202, and the acquisition module 203 is similar to the implementation of the application module 111, the disk 112, and the acquisition module 113 in the architecture diagram shown in fig. 1, and is not described herein again. The functions of the monitoring module 204 are similar to those of the monitoring device 12, and are not described in detail here.
It can be understood that, in the architecture diagrams shown in fig. 1 and 2, since the log processing method is the same, there are problems that the performance analysis efficiency and accuracy are low.
In order to solve the above problems, the technical idea of the technical solution of the present application is as follows: the inventor finds, through practice, that if all call data generated by an application in a period of time are processed in an application module when a QPS of the application is high, for example, call data with the same time consumption generated at the same monitoring point are merged, so that all log data of the same monitoring point in a period of time can be merged into one monitoring log, and thus, when the monitoring log is stored and transmitted to a monitoring device or a monitoring module for processing, the monitoring log is not affected by the QPS, thereby avoiding the occurrence of the above problems 1 to 3, and improving the performance analysis efficiency and the analysis accuracy of the application.
Specifically, the inventor finds that the content of the monitoring log can be optimized and compressed without reducing the data precision and influencing the data analysis result by analyzing the monitoring log and the back-end data storage structure starting from the problem caused by high concurrency. For example, the time-consuming data (i.e., call data) of the requests generated in a period of cache time is cached and counted inside the application module, then the requests are merged and generate a monitoring log, a log thread of the application module is started to refresh the merged monitoring log into a log file of a disk at regular time, correspondingly, the collection module can normally collect the generated monitoring log and report the monitoring log to the monitoring device or the monitoring module, and the monitoring device or the monitoring module adjusts the data statistical calculation method according to the newly merged monitoring log, so that the performance index data statistics can be completed.
According to the technical scheme, the monitoring log generation method and the monitoring log generation device have the advantages that improvement needs to be carried out when the application module generates the monitoring log, real-time monitoring data requested by the application module are cached, the format of the monitoring log is updated, the transmission mode of the data is changed from synchronous output to asynchronous output, time is changed by space, the data volume of the generated monitoring log is reduced, the frequency of refreshing the monitoring log to a disk is reduced, the output volume and the refreshing frequency of the log are controlled, and stable data acquisition of the acquisition module is guaranteed. In the scheme, the monitoring processing terminal needs to be upgraded so as to be compatible with the new log data format, and performance indexes can be calculated for the new log data format, for example, average, maximum, minimum and TP values are calculated, so that the monitoring log quantity can be reduced, and the accurate real-time statistical analysis of the monitoring data is ensured.
As can be seen from the above analysis, an embodiment of the present application provides a method for processing monitoring logs, where multiple pieces of call data generated at each monitoring point when a target application calls a specified method in each caching period are obtained, at least one piece of call information corresponding to the multiple pieces of call data is sequentially stored in a corresponding caching queue according to a monitoring point identifier and a caching period identifier carried by each piece of call data, each piece of call information includes a call execution time and a call execution frequency of the call execution time, and a duration corresponding to each caching period is processed by all pieces of call information in each caching queue, so that multiple pieces of monitoring logs for the target application can be generated and transmitted to a monitoring device, and thus the monitoring device can respectively cache the multiple pieces of monitoring logs according to a data storage template preset at each monitoring point, and processing the cached multiple monitoring logs according to a preset analysis frequency, determining the performance index of the application server corresponding to the target application, wherein the performance index is not influenced by the QPS of the target application, and improving the performance analysis efficiency and the analysis accuracy of the application.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Fig. 3 is a schematic flowchart of a monitoring log processing method according to a first embodiment of the present disclosure. The method is illustrated by information interaction between the application server and the monitoring device in the architecture diagram shown in fig. 1. As shown in fig. 3, the monitoring log processing method may include the steps of:
s301, the application server obtains a plurality of pieces of calling data generated at each monitoring point when the target application calls the specified method in each caching time period.
Each piece of calling data carries a monitoring point identifier and a cache time period identifier.
In this embodiment, when the performance of the target application needs to be monitored, the application server may calculate the performance of the target application when calling the specified method. Alternatively, the specified method is typically one of a number of important methods that the target application performs in actual use.
For example, before a given method is called, monitoring stubs may be placed at a number of different locations, forming a number of different monitoring points. Therefore, when the target application calls the designation method, multiple pieces of call data corresponding to each monitoring point are generated on the application server side, and in addition, in order to facilitate processing of the call data, the application server in this embodiment may obtain the call data generated at each monitoring point based on a set cache time period. Therefore, the application server can acquire pieces of call data generated at each monitoring point when the target application calls the specified method in each cache period.
It can be understood that, in this embodiment, since the target application may call the specified method in different caching time periods, and the call data at different monitoring points may be different, each piece of call data acquired by the application server carries the monitoring point identifier and the caching time period identifier.
S302, the application server stores at least one piece of calling information corresponding to a plurality of pieces of calling data into a corresponding cache queue in sequence according to the monitoring point identifier and the cache time period identifier carried by each piece of calling data.
Optionally, each piece of calling information includes: and each cache queue is named by a monitoring point identifier and a cache time period identifier.
In this embodiment, the application server may store the acquired pieces of call data respectively based on the monitoring point to which the call data belongs and the cache time period to which the call data belongs.
Specifically, the application server firstly sets a cache queue according to a monitoring point identifier and a cache time period identifier carried in the calling data, then analyzes each piece of calling data in the plurality of pieces of calling data, determines calling execution time (called time consumption) carried in the plurality of pieces of calling data and the occurrence frequency (called execution frequency) of each calling execution time, then generates a piece of calling information based on one calling execution time and the calling execution frequency of the calling execution time, and correspondingly, can generate a plurality of pieces of calling information according to the number of the calling execution times carried in the plurality of pieces of calling data; and finally, storing each piece of calling information into a cache queue corresponding to the corresponding monitoring point identifier and the corresponding cache time period identifier.
S303, processing all calling information in each cache queue by the application server at intervals of the duration corresponding to the cache time period to generate a plurality of monitoring logs for the target application.
In this embodiment, since the application server obtains the multiple pieces of call data generated at each monitoring point when the target application calls the designated method once respectively at the time length corresponding to the cache time period, the frequency of caching the call information in the cache queues is the time length corresponding to each interval of the cache time period, so that the application server can process all the call information in each cache queue at the time length corresponding to each interval of the cache time period, generate one monitoring log according to all the call information in each cache queue, and generate multiple monitoring logs of the target application in each cache time period.
It can be understood that, since each cache queue is named by the monitor point identifier and the cache time period identifier, each cache queue corresponds to a unique cache time period and a unique monitor point, thereby ensuring that each monitor point corresponds to a monitor log in each cache time period, which is not affected by the target application calling a specified method (i.e., QPS), and ensuring the timeliness and efficiency of data processing.
S304, the application server sends the plurality of monitoring logs to the monitoring equipment.
In a possible design of the embodiment of the application, the application server may transmit the obtained multiple monitoring logs to the monitoring device for analysis and processing, so as to obtain the performance index of the specified method.
S305, the monitoring device caches a plurality of monitoring logs received from the application server according to a data storage template preset by each monitoring point.
Optionally, a data storage template may be preset in the monitoring device, when the application server sends a plurality of monitoring logs to the monitoring device, the monitoring device may receive the plurality of monitoring logs of the target application, and after receiving the plurality of monitoring logs, may cache the plurality of monitoring logs based on the data storage template, thereby facilitating subsequent data analysis.
Optionally, the data format of the data storage template may be that a combined identifier of the monitoring point identifier and the cache time period identifier is used as a first-level directory, a host name is used as a second-level directory, and call information including the call execution time and the call execution times of the call execution time is provided under the host name. Therefore, the monitoring device may sequentially process the plurality of monitoring logs and then respectively cache the monitoring logs to the corresponding directories according to the data format of the data storage template.
S306, the monitoring equipment processes the cached multiple monitoring logs according to the preset analysis frequency and determines the performance index of the target application corresponding to the application server.
Optionally, the monitoring device may store a preset analysis frequency of each monitoring point, and the preset analysis frequencies of different monitoring points may be the same or different. For example, the preset analysis frequency may be 30 seconds, 1 minute, 2 minutes, or other time duration, and the actual preset analysis frequency of different monitoring points may be set according to actual requirements, which is not described herein again.
For example, in practical application, the monitoring device may directly process the received multiple monitoring logs, or may first cache the received monitoring logs according to the implementation in S305, and then perform performance analysis on the cached multiple monitoring logs according to a preset analysis frequency in the monitoring device, so as to determine a performance index of the specified method, and further determine the performance index of the target application.
In practical applications, the index for evaluating the application performance may include: maximum, minimum, average, TP values, etc. of the execution time are invoked. In this embodiment, the monitoring device may sort all the call execution times under each combined identifier of the monitoring point identifier and the cache time period identifier in a descending order, so as to determine a maximum value and a minimum value of the call execution times; then, using a formula sum (call execution time + number of call executions)/sum (number of call executions), an average value of the call execution times can be calculated; finally, a TP value is calculated, for example, TP99, the monitoring device sorts the call execution time from large to small, and then the sum (call execution times) × 99% value is used as the call execution times (99); for all the call execution times sequenced from large to small, firstly, for the first call execution time, calculating the difference of sum (the number of call execution times) minus the number of call execution times (99), if the difference is less than or equal to the number of call execution times corresponding to the first call execution time, taking the first call execution time as TP99, otherwise, continuing to calculate for the second call execution time until the condition is met.
For example, assume that a group of call execution times and call execution times of a monitoring point in a certain cache time period are: 2 is 50 ms; 54ms is 4; 68ms: 5; 70ms: 6; 3 is 100 ms. If the calculation is performed according to the algorithm of the embodiment of the application, the call execution times are firstly sorted from large to small, namely 100ms:3, 70ms:6, 68ms:5, 54ms:4 and 50ms:2, and then TP99 and TP50 are calculated based on the order of the call execution times from large to small. Specifically, for TP99, count (99) ═ sum (count) 99%, (2+4+5+6+3) × 99%, (19.8), execution time for the first call 100ms, at which time, sum1(count) is 2+4+5+6+3 ═ 20, and 20-19.8 is 0.2; since 0.2 is smaller than the count value 3 corresponding to the call execution time 100ms, the value of TP99 is 100 ms; similarly, for TP50, count (50) ═ sum (count) × 50% > (2+4+5+6+3) × 50% > (10); for the first call execution time 100ms, sum1(count) is 2+4+5+6+3 equal to 20, and 20-10 equal to 10, since 10 is not less than the call execution time 3 corresponding to 100ms, sum2(count) is 2+4+5+6 equal to 17, 17-10 equal to 7, since 7 is not less than the call execution time 6 corresponding to 70ms, sum3(count) is 11, 11-10 equal to 1, 1 is less than the call execution time 5 corresponding to 68ms, and TP50 equal to 68 ms. The other methods for calculating the TP value are similar and will not be described herein.
The monitoring log processing method provided by the embodiment of the application sequentially stores at least one piece of calling information corresponding to a plurality of pieces of calling data into a corresponding cache queue by acquiring a plurality of pieces of calling data generated at each monitoring point when a target application calls a specified method in each cache time period according to a monitoring point identifier and a cache time period identifier carried by each piece of calling data, wherein each piece of calling information comprises a calling execution time and the calling execution times of the calling execution time, and the duration corresponding to each cache time period is processed by processing all pieces of calling information in each cache queue, so that a plurality of monitoring logs aiming at the target application can be generated and transmitted to monitoring equipment, and thus the monitoring equipment can respectively cache the plurality of monitoring logs according to a data storage template preset by each monitoring point and analyze frequency according to the preset, and processing the cached multiple monitoring logs, determining the performance index of the application server corresponding to the target application, wherein the performance index is not influenced by the QPS of the target application, and improving the performance analysis efficiency and the analysis accuracy of the application.
On the basis of the foregoing embodiment, fig. 4 is a schematic flowchart of a second embodiment of the monitoring log processing method provided by the present application. The method is explained with an application server as an execution subject. As shown in fig. 4, the above S302 may be implemented by the following steps:
s401, dividing the obtained calling data into at least one data set according to the monitoring point identification and the cache time period identification carried by each calling data.
Wherein each data set comprises at least one piece of calling data with the same monitoring point identification and the same cache time period identification.
For example, in order to reduce the size of the transmitted data volume, when the application server caches the acquired pieces of call data, the pieces of call data are firstly grouped according to the duration of the caching time period. In general, in order to ensure the efficiency of subsequent processing, the duration of the buffering period is any value from 5 seconds to 10 seconds, and in general, the duration of the buffering period is an integer. For example, the duration of the buffering period is 10 seconds.
Optionally, the application server may divide the pieces of call data generated between xxxx y month zz dd hours and 00 seconds to xxxx y month zz dd hours and 10 seconds into a plurality of data sets, and each data set is marked with a monitoring point identifier-xxxxxxyzzddmm.
S402, for each data set, processing all the calling data in the data set according to the calling execution time of each calling data, and determining at least one piece of calling information.
For example, for each determined data set, first, according to the call execution time in each piece of call data, in the data set, counting the occurrence number of each call execution time, and generating at least one piece of call information corresponding to each data set. Each piece of call information may include each call execution time and the number of call executions of the call execution time.
It can be understood that, for the call data in each data set, if a plurality of pieces of call data have the same call execution time, the call execution times of the call execution time are accumulated to obtain the association relationship between the call execution time and the call execution times.
For example, for a data set of "ap.v 2. token.gettoken-xxxxyyzzzddmm", its corresponding invocation information may be represented by "invocation execution time: the number of times of execution of the call "is expressed in the form of.
And S403, sequentially storing each piece of calling information into a buffer queue corresponding to the data set.
In this step, the buffer queue corresponding to the data set may be represented by the following form:
optionally, when the name of the data set is ap.v2. token.gettoken-xxxxyzzzddmm, the call information of the data set includes: at least one piece of calling information corresponding to the data set is stored in the buffer queue in the following form at 2:7, 10:12 and 16: 5:
accordingly, the above S303 can be implemented by the following steps:
s404, updating the cache time period identification and the monitoring point identification of each cache queue, at least one calling execution time in the cache queue and the calling execution times of each calling execution time into a preset log template every time corresponding to the cache time period, and generating a plurality of monitoring logs for the target application.
Optionally, the application server stores a preset log template, and the format of the preset log template may be as follows:
it can be understood that, in the preset log template, the cache time period identifier, the monitoring point identifier, the application name, the host name, the call execution time, the call execution times, and the like may all be updated, and the embodiment of the present application does not limit the update.
Correspondingly, updating the information in the cache queue determined in S403, that is, at least one piece of call information corresponding to a plurality of pieces of call data within the cache time period xxxxyyzzddmm, to the preset log template may generate the following monitoring log:
optionally, in a specific implementation, after the application server stores at least one piece of call information corresponding to each of the plurality of pieces of call data into a corresponding cache queue, a log thread may be started, the identifiers of all the monitoring points are taken out from the last cache queue at regular time (that is, at a time corresponding to each interval of the cache time period, for example, 10 seconds), and at least one piece of call information in the current cache queue is taken out and combined to generate the monitoring log.
If the current time is xxxxyyzzddmm13 and the duration of the cache time period is 10 seconds, the cache time period of the current cache queue is identified as xxxxyzzddmm 10, and the cache time period of the last cache queue is identified as queuexxxxyzzddmm 00.
As can be seen from the descriptions in S401 to S404, in this embodiment, each monitoring log corresponds to a unique monitoring point identifier and a unique caching time period identifier, and this monitoring log format ensures that only one piece of monitoring log data is generated by one monitoring point in one caching time period, and the monitoring log format does not increase due to the increase of the QPS of the target application, thereby ensuring stable output of the monitoring log amount. Meanwhile, in practical application, performance index analysis of the specified method only depends on call execution time and call execution times, and combination of multiple pieces of call data into one monitoring log does not affect calculation of the performance index, so that the information degree and accuracy of the log data are guaranteed.
According to the monitoring log processing method provided by the embodiment of the application, a plurality of pieces of acquired calling data are divided into at least one data set according to a monitoring point identifier and a cache time period identifier carried by each piece of calling data, all calling data in the data set are processed according to the calling execution time of each piece of calling data for each data set, at least one piece of calling information is determined, and each piece of calling information is sequentially stored in a cache queue corresponding to the data set, so that the cache time period identifier and the monitoring point identifier of each cache queue, the calling execution time of at least one calling execution time in the cache queue and the calling execution times of each calling execution time are updated to a preset log template every time corresponding to the cache time period, and a plurality of monitoring logs applied to a target are generated. According to the technical scheme, the calling data of the designated method are cached, and the monitoring logs are generated in an asynchronous merging mode, so that the number of the monitoring logs is greatly reduced, the utilization rate of the application server processor is reduced, and the stable operation of the target application is ensured.
Optionally, on the basis of the foregoing embodiment, fig. 5 is an interaction schematic diagram of a third embodiment of the monitoring log processing method provided by the present application. As shown in fig. 5, in this embodiment, before the step S303, the method may further include the following steps:
s501, storing a plurality of monitoring logs of the target application into a disk log file.
In this embodiment, the application server is usually installed with a disk, and the monitoring log generated is cached by using the disk. Optionally, after the application server generates multiple monitoring logs of the target application, the multiple monitoring logs may be written into a disk log file of the disk.
S502, acquiring an increment monitoring log in a disk log file according to a preset acquisition period.
Illustratively, the application server stores a preset acquisition period, and can read an incremental log file stored in a time corresponding to a previous preset acquisition period from a disk log file of a disk at regular time according to the preset acquisition period, thereby ensuring the real-time performance and stability of monitoring log acquisition.
Illustratively, the number of monitoring logs in the target time period can be determined based on the number of monitoring points set in the specified method and the duration of the cache time period. For example, the number of monitoring logs generated by the application server per hour is equal to the number of monitoring points multiplied by (3600 seconds/cache time period), wherein the cache time period is in seconds. In practical application, the larger the QPS of the target application is, the more the repeated call execution time in the call data generated in each cache time period is, the more the call data amount combined in the monitoring log in the present application is, the higher the optimization proportion of the monitoring log is, and the more obvious the reduction of the generated monitoring log amount in comparison with the prior art scheme is.
Accordingly, the above S304 may be replaced by the following steps:
and sending the incremental monitoring log to the monitoring equipment.
In this embodiment, in order to reduce the processing load of the monitoring device, the application server may only obtain the incremental monitoring log from the disk log file of the disk, and then send the acquired incremental monitoring log to the monitoring device.
According to the monitoring log processing method provided by the embodiment of the application, the application server can firstly store a plurality of monitoring logs of the target application into the disk log file, then obtain the increment monitoring log in the disk log file according to the preset acquisition period, and send the increment monitoring log to the monitoring equipment. According to the technical scheme, the information degree of the monitoring log obtained by the monitoring equipment is ensured, the occupied cache space of the monitoring equipment is reduced, and the data processing efficiency is improved while the data processing precision is ensured.
Optionally, on the basis of the foregoing embodiment, fig. 6 is a schematic flowchart of a fourth embodiment of the monitoring log processing method provided by the present application. The method is explained with a monitoring device as the executing subject. As shown in fig. 6, S305 may be implemented by:
s601, analyzing each monitoring log, and determining a monitoring point identifier and a cache time period identifier corresponding to each monitoring log.
In this embodiment, after receiving each monitoring log, the monitoring device first analyzes each monitoring log to determine a monitoring point identifier and a cache time period identifier carried on each monitoring log, and further, the monitoring device may also determine a host name, an application name, and the like corresponding to each monitoring log, which provides a premise for subsequently analyzing the performance of the application.
S602, determining at least one piece of calling information corresponding to each monitoring log according to the monitoring point identifier and the cache time period identifier corresponding to each monitoring log.
Wherein each piece of calling information comprises: a call execution time and a number of call executions of the call execution time.
In this embodiment, since the performance index of the specified method called by the target application is mainly determined by the call execution time and the call execution frequency, and the monitoring log has the monitoring point identifier and the cache time period identifier, the monitoring device may first determine at least one piece of call information corresponding to each monitoring log according to the monitoring point identifier and the cache time period identifier corresponding to each monitoring log, and further determine the call condition of the target application at each monitoring point and in each cache time period, for example, the call execution time and the call execution frequency, thereby laying a foundation for the performance analysis of the subsequent target application.
S603, caching at least one piece of calling information corresponding to each monitoring log based on the data storage template corresponding to each monitoring log.
Optionally, the monitoring device is preconfigured with a data storage template, and a duration of a time period in the data storage template corresponds to a preset analysis frequency, so that the monitoring device needs to process according to the preset analysis frequency after receiving the monitoring log, and the monitoring device can merge the call information in the same analysis time period according to the data storage template when the processing time is not reached.
It can be understood that the processing unit of the monitoring log is the duration of a preset time period, for example, 10 seconds, and the preset analysis frequency of the monitoring device is 1 minute in turn, then the analysis time period corresponding to the preset analysis frequency is the duration of 60 seconds, at this time, the monitoring device may perform merging processing on all the call information which have the same monitoring point identifier and fall within the same preset analysis period.
For example, assuming that the preset analysis frequency of the monitoring point identifier key1 is 1 minute, and the starting time of a certain preset analysis cycle is xxxxyyzzz 1600, the ending time thereof should be xxxxyzzz 1700, at this time, in the above monitoring logs, the monitoring data having the buffering time period identifier falling between xxxxyzzz 1600 and xxxxyzzz 1700 are all buffered in the data queue of key 1-xxyzzz 1600, and at the same time, the monitoring data having the buffering time period identifier falling between xxxxyzzz 1700 and xxxxyzzz 1800 are all buffered in the data queue of key 1-xxyzzz 1800, and the buffering of the monitoring logs is sequentially implemented according to this storage logic.
Based on the above embodiments, when the monitor point is identified as ap.v2.token.gettoken and the processing time period is xxxxyyzz1600, the host names of the calls are 11.12.170.11 and hostname 2. 11.12.170.11, the specified method is called 24 times within the duration of xxxxyyzz1600, wherein the calling execution time is 7 times for 2ms, 12 times for 10ms, and 5 times for 16 ms; the hostname2 specifies that the method is called 12 times in total within the duration of xxxxyyzz1600, wherein 6 times are called for 4ms, 6 times are called for 9ms, and the like. Correspondingly, the data storage format of the monitoring equipment end is as follows:
according to the data cache format of the monitoring device side, the data cache content is the call execution time and the corresponding call execution times, and the cache data volume and the call execution time range of the monitoring device side are not increased due to the increase of the times of calling the specified method by the target application, so that the scheme ensures the stability of the call information without influencing the analysis accuracy, and reduces the resource consumption of a monitoring device processor and the occupation size of the cache space.
It can be understood that the embodiment of the present application may further include a scheme in which the monitoring device displays the analysis result or pushes the analysis result to the display device or the output device for performing an early warning, which is not described in detail in this embodiment.
According to the monitoring log processing method provided by the embodiment of the application, the monitoring equipment analyzes each monitoring log, determines the monitoring point identifier and the cache time period identifier corresponding to each monitoring log, determines at least one piece of calling information corresponding to each monitoring log according to the monitoring point identifier and the cache time period identifier corresponding to each monitoring log, and finally caches the at least one piece of calling information corresponding to each monitoring log based on the data storage template corresponding to each monitoring log. In the technical scheme, the monitoring device is compatible with the monitoring log generated by the application server, and when the QPS of the target application is large, the processing data volume is reduced on the premise of not influencing the analysis precision, the transitional resource consumption is avoided, and the usage amount of the cache space is reduced.
Fig. 7 is a schematic flowchart of a fifth monitoring log processing method according to an embodiment of the present application. The method is described with the application server in the architecture diagram shown in fig. 2 as the execution subject. As shown in fig. 7, the monitoring log processing method may include the steps of:
s701, acquiring a plurality of pieces of calling data generated at each monitoring point when the target application calls the designated method in each caching time period, wherein each piece of calling data carries a monitoring point identifier and a caching time period identifier.
S702, according to the monitoring point identification and the cache time period identification carried by each piece of calling data, sequentially storing at least one piece of calling information corresponding to the plurality of pieces of calling data into a corresponding cache queue.
Wherein each piece of calling information comprises: and each cache queue is named by a monitoring point identifier and a cache time period identifier.
And S703, processing all the calling information in each cache queue by the time length corresponding to each interval cache time period to generate a plurality of monitoring logs for the target application.
In this embodiment, the specific implementations of S701 to S703 are the same as the specific implementations of S301 to S303, and are not described herein again.
S704, processing the multiple monitoring logs according to the preset analysis frequency, and determining the performance index of the target application.
In this embodiment, when the application server is integrated with the monitoring module, that is, has a function of performance analysis, the performance analysis of the target application may be directly implemented in the application server, so that after the application server generates a plurality of monitoring logs of the cache time period, the application server may directly process the plurality of monitoring logs according to the preset analysis frequency to determine the performance index of the target application.
It can be understood that, when the preset analysis frequency is inconsistent with the frequency of the data cache, the generated monitoring log may be stored in a disk log file of the disk first, and then the data is read from the disk at regular time according to the preset analysis frequency and is analyzed.
Optionally, the embodiment of the present application may further include a scheme for displaying the analysis result or pushing the analysis result, which is not described in detail in this embodiment.
For a specific implementation of this step, reference may be made to the scheme that the monitoring device processes the received monitoring log described in each embodiment, and details are not described here again.
According to the monitoring log processing method provided by the embodiment of the application, an application server can acquire a plurality of pieces of calling data generated at each monitoring point when a target application calls a specified method in each caching time period, sequentially stores at least one piece of calling information corresponding to the plurality of pieces of calling data into a corresponding caching queue according to a monitoring point identifier and a caching time period identifier carried by each piece of calling data, processes all pieces of calling information in each caching queue at intervals of time corresponding to the caching time period, generates a plurality of monitoring logs for the target application, and finally processes the plurality of monitoring logs according to a preset analysis frequency to determine a performance index of the target application. In the technical scheme, when the application server is integrated with the function of monitoring log analysis, the purpose of application performance analysis can be achieved, the application performance analysis is not influenced by QPS, interaction between devices is not needed, and the interaction flow is simplified.
The following are embodiments of the apparatus of the present application that may be used to perform embodiments of the method of the present application. For details which are not disclosed in the embodiments of the apparatus of the present application, reference is made to the embodiments of the method of the present application.
Fig. 8 is a schematic structural diagram of a monitoring log processing apparatus according to a first embodiment of the present disclosure. Referring to fig. 8, the apparatus may include:
an obtaining module 801, configured to obtain multiple pieces of call data generated at each monitoring point when a target application calls a specified method in each cache time period, where each piece of call data carries a monitoring point identifier and a cache time period identifier;
a cache module 802, configured to store, in sequence, at least one piece of call information corresponding to the multiple pieces of call data into a corresponding cache queue according to a monitor point identifier and a cache time period identifier carried in each piece of call data, where each piece of call information includes: calling execution time and calling execution times of the calling execution time, wherein each cache queue is named by a monitoring point identifier and a cache time period identifier;
a processing module 803, configured to process all the call information in each cache queue every a duration corresponding to the cache time period, and generate multiple monitoring logs for the target application;
a sending module 804, configured to send the multiple monitoring logs to the monitoring device.
In one possible design of the embodiment of the present application, the cache module 802 is specifically configured to:
dividing the obtained calling data into at least one data set according to a monitoring point identifier and a cache time period identifier carried by each piece of calling data, wherein each data set comprises at least one piece of calling data with the same monitoring point identifier and the same cache time period identifier;
for each data set, processing all the calling data in the data set according to the calling execution time of each calling data, and determining at least one piece of calling information;
and sequentially storing each piece of calling information into a cache queue corresponding to the data set.
Optionally, the processing module 803 is specifically configured to update, every time duration corresponding to the cache time period, the cache time period identifier and the monitoring point identifier of each cache queue, at least one call execution time in the cache queue, and the call execution times of each call execution time to a preset log template, so as to generate multiple monitoring logs for the target application.
In another possible design of this embodiment, the cache module 802 is further configured to process all the call information in each cache queue at a time interval corresponding to the cache time period by the processing module 803, generate multiple monitoring logs for the target application, and store the multiple monitoring logs of the target application in a disk log file;
the obtaining module 801 is further configured to obtain an incremental monitoring log in the disk log file according to a preset collection period;
a sending module 804, configured to send the incremental monitoring log to a monitoring device.
The apparatus provided in the embodiment of the present application may be configured to execute the scheme of the application server in the method embodiments described in fig. 3 to fig. 6, and the implementation principle and the technical effect are similar, which are not described herein again.
Fig. 9 is a schematic structural diagram of a second monitoring log processing apparatus according to the present application. Referring to fig. 9, the apparatus may include:
a receiving module 901, configured to receive multiple monitoring logs of a target application from an application server;
a caching module 902, configured to cache the multiple monitoring logs respectively according to a data storage template preset in each monitoring point;
the processing module 903 is configured to process the multiple cached monitoring logs according to a preset analysis frequency, and determine a performance index of the target application corresponding to the application server.
In a possible design of the embodiment of the present application, the cache module 902 is specifically configured to:
analyzing each monitoring log to determine a monitoring point identifier and a cache time period identifier corresponding to each monitoring log;
determining at least one piece of calling information corresponding to each monitoring log according to the monitoring point identifier and the cache time period identifier corresponding to each monitoring log, wherein each piece of calling information comprises: a call execution time and a number of call executions of said call execution time;
and caching at least one piece of calling information corresponding to each monitoring log based on the data storage template corresponding to each monitoring log.
The apparatus provided in the embodiment of the present application may be configured to execute the scheme of the monitoring device in the method embodiment described in fig. 3 to fig. 6, and the implementation principle and the technical effect are similar, which are not described herein again.
Fig. 10 is a schematic structural diagram of a third monitoring log processing apparatus according to the present application. Referring to fig. 10, the apparatus may include:
an obtaining module 1001, configured to obtain multiple pieces of call data generated at each monitoring point when a target application calls a designated device in each cache time period, where each piece of call data carries a monitoring point identifier and a cache time period identifier;
the cache module 1002 is configured to store at least one piece of calling information corresponding to the multiple pieces of calling data in a corresponding cache queue in sequence according to a monitor point identifier and a cache time period identifier carried by each piece of calling data, where each piece of calling information includes: calling execution time and calling execution times of the calling execution time, wherein each cache queue is named by a monitoring point identifier and a cache time period identifier;
the processing module 1003 is configured to process all the call information in each cache queue at intervals of a duration corresponding to the cache time period, generate multiple monitoring logs for the target application, process the multiple monitoring logs according to a preset analysis frequency, and determine a performance index of the target application.
The apparatus provided in the embodiment of the present application may be configured to execute the scheme in the embodiment of the method illustrated in fig. 7, and the implementation principle and the technical effect are similar, which are not described herein again.
It should be noted that the division of the modules of the above apparatus is only a logical division, and the actual implementation may be wholly or partially integrated into one physical entity, or may be physically separated. And these modules can be realized in the form of software called by processing element; or may be implemented entirely in hardware; and part of the modules can be realized in the form of calling software by the processing element, and part of the modules can be realized in the form of hardware. For example, the processing module may be a separate processing element, or may be integrated into a chip of the apparatus, or may be stored in a memory of the apparatus in the form of program code, and a processing element of the apparatus calls and executes the functions of the above determination module. Other modules are implemented similarly. In addition, all or part of the modules can be integrated together or can be independently realized. The processing element described herein may be an integrated circuit having signal processing capabilities. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in the form of software.
Fig. 11 is a schematic structural diagram of a first embodiment of an application server according to the present application. As shown in fig. 11, the application server may include: the system comprises a processor 1101, a memory 1102, a transceiver 1103 and a system bus 1104, wherein the memory 1102 and the transceiver 1103 are connected with the processor 1101 through the system bus 1104 and complete mutual communication, the memory 1102 is used for storing computer execution instructions, the transceiver 1103 is used for communicating with other devices, and the application server scheme in the embodiment shown in fig. 3 to fig. 6 is realized when the processor 1101 executes the computer execution instructions.
Fig. 12 is a schematic structural diagram of an embodiment of a monitoring device provided in the embodiment of the present application. As shown in fig. 12, the monitoring apparatus may include: the monitoring device comprises a processor 1201, a memory 1202, a transceiver 1203 and a system bus 1204, wherein the memory 1202 and the transceiver 1203 are connected with the processor 1201 through the system bus 1204 and are used for achieving mutual communication, the memory 1202 is used for storing computer execution instructions, the transceiver 1203 is used for communicating with other devices, and the monitoring device scheme in the embodiment shown in the above fig. 3 to fig. 6 is achieved when the processor 1201 executes the computer execution instructions.
Fig. 13 is a schematic structural diagram of a second application server according to an embodiment of the present application. As shown in fig. 13, the application server may include: the system comprises a processor 1301, a memory 1302, a communication interface 1303 and a system bus 1304, wherein the memory 1302 and the transceiver 1303 are connected with the processor 1301 through the system bus 1304 and complete mutual communication, the memory 1302 is used for storing computer execution instructions, the communication interface 1303 is used for communicating with other devices, and the scheme of the application server in the embodiment shown in fig. 7 is realized when the processor 1301 executes the computer execution instructions.
In fig. 10 to 13, the processor may be a general-purpose processor, including a central processing unit CPU, a Network Processor (NP), and the like; but also a digital signal processor DSP, an application specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components.
The memory may comprise Random Access Memory (RAM), read-only memory (RAM), and non-volatile memory (non-volatile memory), such as at least one disk memory.
The transceiver or communication interface is used to enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries).
The system bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The system bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
Optionally, an embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is enabled to execute the scheme of the application server in the method embodiment shown in fig. 3 to 6, or the computer is enabled to execute the scheme of the monitoring device in the method embodiment shown in fig. 3 to 6, or the computer is enabled to execute the scheme of the application server in the method embodiment shown in fig. 7.
Optionally, an embodiment of the present application further provides a chip for executing an instruction, where the chip is configured to execute a scheme of an application server in the method embodiment shown in fig. 3 to 6, or a scheme of a monitoring device in the method embodiment shown in fig. 3 to 6, or a scheme of an application server in the method embodiment shown in fig. 7.
An embodiment of the present application further provides a program product, where the program product includes a computer program, where the computer program is stored in a computer-readable storage medium, and the computer program can be read from the computer-readable storage medium by at least one processor, and when the computer program is executed by the at least one processor, the scheme of the application server in the method embodiment shown in fig. 3 to 6 above, or the scheme of the monitoring device in the method embodiment shown in fig. 3 to 6 above, or the scheme of the application server in the method embodiment shown in fig. 7 above may be implemented.
In the present application, "at least one" means one or more, "a plurality" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone, wherein A and B can be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship; in the formula, the character "/" indicates that the preceding and following related objects are in a relationship of "division". "at least one of the following" or similar expressions refer to any combination of these items, including any combination of the singular or plural items.
It is to be understood that the various numerical references referred to in the embodiments of the present application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of the present application. In the embodiment of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiment of the present application.
Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (18)
1. A monitoring log processing method is characterized by comprising the following steps:
acquiring a plurality of pieces of calling data generated at each monitoring point when a target application calls a specified method in each caching time period, wherein each piece of calling data carries a monitoring point identifier and a caching time period identifier;
according to the monitoring point identification and the cache time period identification carried by each piece of calling data, at least one piece of calling information corresponding to the calling data is sequentially stored into a corresponding cache queue, and each piece of calling information comprises: calling execution time and calling execution times of the calling execution time, wherein each cache queue is named by a monitoring point identifier and a cache time period identifier;
processing all calling information in each cache queue at intervals of a time length corresponding to the cache time period to generate a plurality of monitoring logs for the target application;
and sending the monitoring logs to a monitoring device.
2. The method according to claim 1, wherein the sequentially storing at least one piece of calling information corresponding to the plurality of pieces of calling data in a corresponding cache queue according to a monitoring point identifier and a cache time period identifier carried by each piece of calling data comprises:
dividing the obtained calling data into at least one data set according to a monitoring point identifier and a cache time period identifier carried by each piece of calling data, wherein each data set comprises at least one piece of calling data with the same monitoring point identifier and the same cache time period identifier;
for each data set, processing all the calling data in the data set according to the calling execution time of each calling data, and determining at least one piece of calling information;
and sequentially storing each piece of calling information into a cache queue corresponding to the data set.
3. The method according to claim 2, wherein the processing all call information in each cache queue every time interval corresponding to the cache time period to generate a plurality of monitoring logs for the target application comprises:
and updating the cache time period identification and the monitoring point identification of each cache queue, at least one calling execution time in the cache queue and the calling execution times of each calling execution time into a preset log template every other time length corresponding to the cache time period, and generating a plurality of monitoring logs for the target application.
4. The method according to any one of claims 1 to 3, wherein after processing all call information in each cache queue at a time corresponding to the interval of the cache time period and generating a plurality of monitoring logs for the target application, the method further comprises:
storing a plurality of monitoring logs of the target application into a disk log file;
acquiring an increment monitoring log in the disk log file according to a preset acquisition period;
the sending the plurality of monitoring logs to the monitoring device includes:
and sending the incremental monitoring log to a monitoring device.
5. A monitoring log processing method is characterized by comprising the following steps:
receiving a plurality of monitoring logs of a target application from an application server;
caching the multiple monitoring logs respectively according to a data storage template preset by each monitoring point;
and processing the plurality of cached monitoring logs according to a preset analysis frequency, and determining the performance index of the target application corresponding to the application server.
6. The method according to claim 5, wherein the caching the plurality of monitoring logs according to a data storage template preset for each monitoring point respectively comprises:
analyzing each monitoring log to determine a monitoring point identifier and a cache time period identifier corresponding to each monitoring log;
determining at least one piece of calling information corresponding to each monitoring log according to the monitoring point identifier and the cache time period identifier corresponding to each monitoring log, wherein each piece of calling information comprises: a call execution time and a number of call executions of said call execution time;
and caching at least one piece of calling information corresponding to each monitoring log based on the data storage template corresponding to each monitoring log.
7. A monitoring log processing method is characterized by comprising the following steps:
acquiring a plurality of pieces of calling data generated at each monitoring point when a target application calls a specified method in each caching time period, wherein each piece of calling data carries a monitoring point identifier and a caching time period identifier;
according to the monitoring point identification and the cache time period identification carried by each piece of calling data, at least one piece of calling information corresponding to the calling data is sequentially stored into a corresponding cache queue, and each piece of calling information comprises: calling execution time and calling execution times of the calling execution time, wherein each cache queue is named by a monitoring point identifier and a cache time period identifier;
processing all calling information in each cache queue at intervals of a time length corresponding to the cache time period to generate a plurality of monitoring logs for the target application;
and processing the plurality of monitoring logs according to a preset analysis frequency to determine the performance index of the target application.
8. A monitoring log processing apparatus, comprising:
the acquisition module is used for acquiring a plurality of pieces of calling data generated at each monitoring point when the target application calls the specified method in each cache time period, and each piece of calling data carries a monitoring point identifier and a cache time period identifier;
the cache module is used for sequentially storing at least one piece of calling information corresponding to the calling data into a corresponding cache queue according to a monitoring point identifier and a cache time period identifier carried by each piece of calling data, and each piece of calling information comprises: calling execution time and calling execution times of the calling execution time, wherein each cache queue is named by a monitoring point identifier and a cache time period identifier;
the processing module is used for processing all calling information in each cache queue at intervals of time corresponding to the cache time period to generate a plurality of monitoring logs for the target application;
and the sending module is used for sending the monitoring logs to the monitoring equipment.
9. The apparatus of claim 8, wherein the cache module is specifically configured to:
dividing the obtained calling data into at least one data set according to a monitoring point identifier and a cache time period identifier carried by each piece of calling data, wherein each data set comprises at least one piece of calling data with the same monitoring point identifier and the same cache time period identifier;
for each data set, processing all the calling data in the data set according to the calling execution time of each calling data, and determining at least one piece of calling information;
and sequentially storing each piece of calling information into a cache queue corresponding to the data set.
10. The apparatus according to claim 9, wherein the processing module is specifically configured to update, every time duration corresponding to the cache time period, a cache time period identifier and a monitoring point identifier of each cache queue, at least one call execution time in the cache queues, and the number of call execution times of each call execution time to a preset log template, so as to generate multiple monitoring logs for the target application.
11. The apparatus according to any one of claims 8 to 10, wherein the cache module is further configured to process all the call information in each cache queue at a time interval corresponding to the cache time period by the processing module, generate multiple monitoring logs for the target application, and store the multiple monitoring logs of the target application in a disk log file;
the acquisition module is further used for acquiring the incremental monitoring logs in the disk log file according to a preset acquisition period;
the sending module is specifically configured to send the incremental monitoring log to a monitoring device.
12. A monitoring log processing apparatus, comprising:
the receiving module is used for receiving a plurality of monitoring logs of the target application from the application server;
the cache module is used for caching the plurality of monitoring logs respectively according to a data storage template preset by each monitoring point;
and the processing module is used for processing the plurality of cached monitoring logs according to a preset analysis frequency and determining the performance index of the target application corresponding to the application server.
13. The apparatus of claim 12, wherein the cache module is specifically configured to:
analyzing each monitoring log to determine a monitoring point identifier and a cache time period identifier corresponding to each monitoring log;
determining at least one piece of calling information corresponding to each monitoring log according to the monitoring point identifier and the cache time period identifier corresponding to each monitoring log, wherein each piece of calling information comprises: a call execution time and a number of call executions of said call execution time;
and caching at least one piece of calling information corresponding to each monitoring log based on the data storage template corresponding to each monitoring log.
14. A monitoring log processing apparatus, comprising:
the acquisition module is used for acquiring a plurality of pieces of calling data generated at each monitoring point when the target application calls the designated device in each cache time period, and each piece of calling data carries a monitoring point identifier and a cache time period identifier;
the cache module is used for sequentially storing at least one piece of calling information corresponding to the calling data into a corresponding cache queue according to a monitoring point identifier and a cache time period identifier carried by each piece of calling data, and each piece of calling information comprises: calling execution time and calling execution times of the calling execution time, wherein each cache queue is named by a monitoring point identifier and a cache time period identifier;
and the processing module is used for processing all calling information in each cache queue at intervals of time corresponding to the cache time period, generating a plurality of monitoring logs for the target application, processing the plurality of monitoring logs according to a preset analysis frequency and determining the performance index of the target application.
15. An application server comprising a processor, a memory, a transceiver, and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1-4 when executing the program.
16. A monitoring device comprising a processor, a memory, a transceiver and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of claim 5 or 6 when executing the program.
17. An application server comprising a processor, a memory and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to claim 7 when executing the program.
18. A computer-readable storage medium having stored thereon computer-executable instructions for performing the method of any one of claims 1-4 when executed by a processor; or
The computer executable instructions when executed by a processor are for implementing the method of claim 5 or 6 as described above; or
The computer executable instructions when executed by a processor are for implementing the method as claimed in claim 7 above.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011271126.4A CN113760640A (en) | 2020-11-13 | 2020-11-13 | Monitoring log processing method, device, equipment and storage medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202011271126.4A CN113760640A (en) | 2020-11-13 | 2020-11-13 | Monitoring log processing method, device, equipment and storage medium |
Publications (1)
Publication Number | Publication Date |
---|---|
CN113760640A true CN113760640A (en) | 2021-12-07 |
Family
ID=78786007
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202011271126.4A Pending CN113760640A (en) | 2020-11-13 | 2020-11-13 | Monitoring log processing method, device, equipment and storage medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113760640A (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114579269A (en) * | 2022-02-08 | 2022-06-03 | 阿里巴巴(中国)有限公司 | Task scheduling method and device |
CN117478535A (en) * | 2023-12-26 | 2024-01-30 | 北京天空卫士网络安全技术有限公司 | Log storage method and device |
CN118689859A (en) * | 2024-08-29 | 2024-09-24 | 山东云海国创云计算装备产业创新中心有限公司 | Log landing method and device, storage medium and electronic equipment |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140082184A1 (en) * | 2012-09-14 | 2014-03-20 | Alibaba Group Holding Limited | Method and system for monitoring execution of user request in distributed system |
CN107992398A (en) * | 2017-12-22 | 2018-05-04 | 宜人恒业科技发展(北京)有限公司 | The monitoring method and monitoring system of a kind of operation system |
CN108509323A (en) * | 2018-01-30 | 2018-09-07 | 深圳壹账通智能科技有限公司 | Method for processing business, device based on log analysis and computer equipment |
CN109263657A (en) * | 2018-08-31 | 2019-01-25 | 北京图森未来科技有限公司 | A kind of long-distance monitoring method of automatic driving vehicle, device and system |
CN109388542A (en) * | 2018-10-26 | 2019-02-26 | 北京百悟科技有限公司 | A kind of monitoring method and device, computer readable storage medium |
CN109617869A (en) * | 2018-12-06 | 2019-04-12 | 中铁程科技有限责任公司 | Inter-network log real-time collecting method and terminal |
CN110309130A (en) * | 2018-03-21 | 2019-10-08 | 中国人民财产保险股份有限公司 | A kind of method and device for host performance monitor |
-
2020
- 2020-11-13 CN CN202011271126.4A patent/CN113760640A/en active Pending
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140082184A1 (en) * | 2012-09-14 | 2014-03-20 | Alibaba Group Holding Limited | Method and system for monitoring execution of user request in distributed system |
CN107992398A (en) * | 2017-12-22 | 2018-05-04 | 宜人恒业科技发展(北京)有限公司 | The monitoring method and monitoring system of a kind of operation system |
CN108509323A (en) * | 2018-01-30 | 2018-09-07 | 深圳壹账通智能科技有限公司 | Method for processing business, device based on log analysis and computer equipment |
CN110309130A (en) * | 2018-03-21 | 2019-10-08 | 中国人民财产保险股份有限公司 | A kind of method and device for host performance monitor |
CN109263657A (en) * | 2018-08-31 | 2019-01-25 | 北京图森未来科技有限公司 | A kind of long-distance monitoring method of automatic driving vehicle, device and system |
CN109388542A (en) * | 2018-10-26 | 2019-02-26 | 北京百悟科技有限公司 | A kind of monitoring method and device, computer readable storage medium |
CN109617869A (en) * | 2018-12-06 | 2019-04-12 | 中铁程科技有限责任公司 | Inter-network log real-time collecting method and terminal |
Non-Patent Citations (1)
Title |
---|
张计飞;: "通用应用日志监管平台", 中国管理信息化, no. 02, 15 January 2017 (2017-01-15) * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114579269A (en) * | 2022-02-08 | 2022-06-03 | 阿里巴巴(中国)有限公司 | Task scheduling method and device |
CN117478535A (en) * | 2023-12-26 | 2024-01-30 | 北京天空卫士网络安全技术有限公司 | Log storage method and device |
CN117478535B (en) * | 2023-12-26 | 2024-04-19 | 北京天空卫士网络安全技术有限公司 | Log storage method and device |
CN118689859A (en) * | 2024-08-29 | 2024-09-24 | 山东云海国创云计算装备产业创新中心有限公司 | Log landing method and device, storage medium and electronic equipment |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN113760640A (en) | Monitoring log processing method, device, equipment and storage medium | |
CN109947668A (en) | The method and apparatus of storing data | |
CN106209974B (en) | A kind of method of data synchronization, equipment and system | |
US11645179B2 (en) | Method and apparatus of monitoring interface performance of distributed application, device and storage medium | |
CN103067297A (en) | Dynamic load balancing method and device based on resource consumption prediction | |
CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
CN111159219B (en) | Data management method, device, server and storage medium | |
CN111966289A (en) | Partition optimization method and system based on Kafka cluster | |
CN113468226A (en) | Service processing method, device, electronic equipment and storage medium | |
CN111782692A (en) | Frequency control method and device | |
CN111522870A (en) | Database access method, middleware and readable storage medium | |
CN114398520A (en) | Data retrieval method, system, device, electronic equipment and storage medium | |
CN113422808A (en) | Internet of things platform HTTP information pushing method, system, device and medium | |
US10983888B1 (en) | System and method for generating dynamic sparse exponential histograms | |
CN111953567A (en) | Method, system, equipment and medium for configuring multi-cluster management software parameters | |
EP4357931A1 (en) | Shard adjustment method and apparatus for time series database, device, and readable storage medium | |
WO2022266975A1 (en) | Method for millisecond-level accurate slicing of time series stream data | |
CN112131267B (en) | Count processing method, device, server and count processing system | |
CN115269519A (en) | Log detection method and device and electronic equipment | |
CN114722264A (en) | Data query method and device, electronic equipment and storage medium | |
CN108009849B (en) | Method and device for generating account state | |
CN113791904B (en) | Method, apparatus, device and readable storage medium for processing query input | |
CN113225228B (en) | Data processing method and device | |
CN113282405B (en) | Load adjustment optimization method and terminal | |
CN117609315B (en) | Data processing method, device, equipment and readable storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |