CN112988516A - Method for realizing global service monitoring based on log enhancement - Google Patents
Method for realizing global service monitoring based on log enhancement Download PDFInfo
- Publication number
- CN112988516A CN112988516A CN202110293088.0A CN202110293088A CN112988516A CN 112988516 A CN112988516 A CN 112988516A CN 202110293088 A CN202110293088 A CN 202110293088A CN 112988516 A CN112988516 A CN 112988516A
- Authority
- CN
- China
- Prior art keywords
- log
- data
- service
- enhancement
- performance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 41
- 238000000034 method Methods 0.000 title claims abstract description 22
- 239000000523 sample Substances 0.000 claims abstract description 19
- 230000006399 behavior Effects 0.000 claims description 3
- 238000004140 cleaning Methods 0.000 claims description 3
- 238000013480 data collection Methods 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 3
- 238000004458 analytical method Methods 0.000 description 3
- 238000012423 maintenance Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/324—Display of status information
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method for realizing global service monitoring based on log enhancement, which comprises the following steps: s1: configuring a Java log enhancement file in an application performance monitoring program, and improving a probe into a log enhancement probe; s2: monitoring the application system by using the application performance monitoring program in the step S1; s3: when the monitored application system is started, the application performance monitoring system puts the log enhancement probe into the monitored application system; s4: the log platform collects enhanced logs generated by the monitored system; s5: and the log platform arranges the enhanced logs and performs associated display on the service data and the performance data. The invention is based on Java byte code technology, enhances log output frame, adds user ID, Url and other service related data when outputting log, and performs relevance output on performance data and service data after collecting log data through log platform, and further displays the performance data in service angle through page.
Description
Technical Field
The invention relates to a service monitoring method, in particular to a method for realizing global service monitoring based on log enhancement.
Background
With the popularization of the internet, more and more services are carried out through the internet, and the performance of applications is greatly tested by more and more users, so that the design of application architectures is more and more complicated, the performance monitoring of application systems becomes more and more important, after all, the experience of users is very poor when the users handle services on the service systems with poor performance, and the handling efficiency is greatly influenced.
The initial performance monitoring scheme can only monitor through dial testing and personal feelings of users, and the mode is easily influenced by subjective awareness and network conditions of the users, and performance feedback of applications is not objective, accurate and timely. Therefore, an application performance monitoring system such as Pinpoint and Sky Walking is promoted, a probe is implanted when an application is started through a class enhancement mechanism of a Java language, performance data such as application response time, throughput, sql execution time, external call time consumption, application memory and CPU use conditions are collected and sent to a server through a network for sorting and displaying, so that the application performance monitoring can be displayed more intuitively through a graphical form, and system management and operation and maintenance personnel can know the application performance condition in time. The defects of the existing application monitoring system are as follows: 1. performance data displayed by monitoring of application performance is too technical, so that the performance data is not convenient for users to understand, and the application performance cannot be monitored from a service perspective; 2. the existing log generally lacks service related information, and the service related information needs to be acquired by modifying codes in application performance monitoring, so that the workload is large. Therefore, the prior art has yet to be improved.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a method for realizing global service monitoring based on log enhancement, which is characterized in that on the basis of a traditional application performance monitoring system probe, service related information is added in originally output log contents by enhancing log frame related classes, and the global monitoring is realized from the service perspective.
The technical scheme adopted by the invention for solving the technical problems is to provide a method for realizing global service monitoring based on log enhancement, which comprises the following steps: s1: configuring a Java log enhancement file in an application performance monitoring program, and improving a probe into a log enhancement probe; s2: monitoring the application system by using the application performance monitoring program in the step S1; s3: when the monitored application system is started, the application performance monitoring system puts the log enhancement probe into the monitored application system; s4: the log platform collects enhanced logs generated by the monitored system; s5: and the log platform arranges the enhanced logs and performs associated display on the service data and the performance data.
Further, in the step S1, the application performance monitoring program adopts Sky Walking, adds service data to be collected in the Java log enhancement file, and associates the service data with the performance data.
Furthermore, the Java log enhancement file is a class file, a Java code is compiled according to the service data required to be collected and the incidence relation between the service data and the performance data, and the code is compiled into the class file required by the Java virtual machine through a Java compiler.
Further, in step S3, the application performance monitoring system puts the log enhancement probe into the monitored application system, that is, loads the class file of the Java log enhancement file into the monitored application system, and loads the class file into the memory space of the monitored application system and runs after the class file is loaded by the class loader, verified by the bytecode verifier and interpreted by the interpreter in sequence.
Furthermore, after the class file is loaded by the monitored application system, the log generation file is dynamically modified according to the service data to be acquired by the class file and the incidence relation between the service data and the performance data, and the log generation file code logic is changed to increase the incidence logic between the service data acquisition and the service data and the performance data, so that the generated enhanced log comprises the service data and the incidence relation between the service data and the performance data.
Further, the step S4 includes that the log platform performs data cleaning on the collected log data according to a regular expression cutting template, and then performs Search and storage on the log data through an API provided by Elastic Search; and according to the service types, the query sentences of each service scene are obtained through sorting, and a service data query unit for querying through log keywords, reports and user behaviors is formed.
Further, the display page of the log platform in the step S5 is provided with a query entry, and the query entry invokes the service data query unit to perform service performance state query through an Elastic Search engine.
Compared with the prior art, the invention has the following beneficial effects: the method for realizing global service monitoring based on log enhancement provided by the invention is based on Java byte code technology, enhances a log output frame, adds service related data such as user ID, Url and the like when outputting logs, performs relevance output on performance data and service data after collecting log data through a log platform, and further displays the performance data in a service angle through a page, so that the performance of the service can be displayed more intuitively in a graphical form, and users and operation and maintenance personnel can know the performance condition of application in time.
Drawings
FIG. 1 is a flowchart of a method for implementing global service monitoring based on log enhancement in an embodiment of the present invention;
FIG. 2 is a block diagram of a method for implementing global service monitoring based on log enhancement according to an embodiment of the present invention;
fig. 3 is a flowchart illustrating the loading of the journal enhancement probe according to an embodiment of the present invention.
Detailed Description
The invention is further described below with reference to the figures and examples.
FIG. 1 is a flowchart of a method for implementing global service monitoring based on log enhancement in an embodiment of the present invention; fig. 2 is an architecture diagram of a method for implementing global service monitoring based on log enhancement in the embodiment of the present invention.
Referring to fig. 1 and fig. 2, a method for implementing global service monitoring based on log enhancement according to an embodiment of the present invention includes the following steps:
s1: configuring a Java log enhancement file in an application performance monitoring program, and improving a probe into a log enhancement probe;
s2: monitoring the application system by using the application performance monitoring program in the step S1;
s3: when the monitored application system is started, the application performance monitoring system puts the log enhancement probe into the monitored application system;
s4: the log platform collects enhanced logs generated by the monitored system;
the log platform carries out data cleaning on the collected log data according to a regular expression cutting template, and then carries out searching and storage on the log data through an API (application program interface) provided by Elastic Search; and according to the service types, the query sentences of each service scene are obtained through sorting, and a service data query unit for querying through log keywords, reports and user behaviors is formed.
S5: and the log platform arranges the enhanced logs and performs associated display on the service data and the performance data.
Specifically, the application performance monitoring program adds service data to be collected in a Java log enhancement file by using Sky Walking, and associates the service data with the performance data.
And the Sky Walking APM tool comprises distributed tracking, performance index analysis, application and service dependency analysis and the like. The core of the Sky Walking is a storage platform of data analysis and measurement results, analysis and measurement data are submitted to the Sky Walking collector in an HTTP or GRPC mode, and the Sky Walking supports data collection from multiple sources and multiple formats: data formats for Sky Walking Agent, Zipkin v1/v2, Istio survey, Envoy metric, etc. in multiple languages.
The reason why the Sky Walking is selected for the journal enhancement probe is that according to the contrast, the performance consumption is small, namely, the influence on the application is low, and is within three percent and almost negligible. In addition, the system supports Elastic Search for storage, is consistent with the storage scheme of most log systems, is convenient for data storage, and has high query efficiency.
And a display page of the log platform is provided with a query inlet, the query inlet calls a service data query unit, and service performance state query is carried out through an Elastic Search engine.
Elastic Search is a Lucene-based Search server. It provides a distributed multi-user capability full-text search engine based on Restful Web interface. Elastic Search was developed in Java and released as open source code under the Apache licensing terms, and is currently a popular enterprise-level Search engine. The design is used in cloud computing, can achieve real-time search, and is stable, reliable, quick, and convenient to install and use. Elastic Search is the most popular enterprise Search engine as shown by DB-Engineers' ranking. Based on Elastic Search, the collected log data can be stored quickly and permanently, and meanwhile, the data which needs to be searched by the user can be inquired quickly and presented.
Referring to fig. 3, in the method for implementing global service monitoring based on log enhancement according to the embodiment of the present invention, the Java log enhancement file is a class file, a Java code is compiled according to service data to be acquired and an association relationship between the service data and performance data, and the code is compiled into the class file required by the Java virtual machine through a Java compiler.
Specifically, in step S3, the application performance monitoring system puts the log enhancement probe into the monitored application system, that is, loads the class file of the Java log enhancement file into the monitored application system, and loads the class file into the memory space of the monitored application system and runs the class file after being loaded by the class loader, verified by the bytecode verifier and interpreted by the interpreter in sequence.
Specifically, after the class file is loaded by the monitored application system, the log generation file is dynamically modified according to the service data to be acquired by the class file and the association relationship between the service data and the performance data, and the log generation file code logic is changed to increase the service data acquisition and the association logic between the service data and the performance data, so that the generated enhanced log comprises the service data and the association relationship between the service data and the performance data.
In summary, the method for realizing global service monitoring based on log enhancement in the embodiments of the present invention is based on Java bytecode technology, and enhances a log output framework, adds service-related data such as user ID, Url, and the like when outputting logs, and performs relevance output on performance data and service data after collecting log data by a log platform, and further performs presentation of the performance data from a service angle through a page, so that the performance of a service can be presented more intuitively in a graphical form, and a user and operation and maintenance staff can know performance conditions of an application in time.
Although the present invention has been described with respect to the preferred embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (7)
1. A method for realizing global service monitoring based on log enhancement is characterized by comprising the following steps:
s1: configuring a Java log enhancement file in an application performance monitoring program, and improving a probe into a log enhancement probe;
s2: monitoring the application system by using the application performance monitoring program in the step S1;
s3: when the monitored application system is started, the application performance monitoring system puts the log enhancement probe into the monitored application system;
s4: the log platform collects enhanced logs generated by the monitored system;
s5: and the log platform arranges the enhanced logs and performs associated display on the service data and the performance data.
2. The method for implementing global service monitoring based on log enhancement as claimed in claim 1, wherein the application performance monitoring program in step S1 adopts Sky Walking to add service data to be collected in the Java log enhancement file and associate the service data with the performance data.
3. The method for realizing global service monitoring based on log enhancement as claimed in claim 2, wherein the Java log enhancement file is a class file, Java code is compiled according to service data to be collected and an association relationship between the service data and performance data, and the code is compiled into the class file required by the Java virtual machine through a Java compiler.
4. The method according to claim 3, wherein the application performance monitoring system in step S3 inserts the log enhancement probe into the monitored application system, that is, loads the class file of the Java log enhancement file into the monitored application system, and loads the class file into the memory space of the monitored application system and runs the class file after being loaded by the class loader, verified by the bytecode verifier and interpreted by the interpreter in sequence.
5. The method for realizing global service monitoring based on log enhancement as claimed in claim 4, wherein after the class file is loaded by the monitored application system, the log generation file is dynamically modified according to the service data to be collected by the class file and the association relationship between the service data and the performance data, and the log generation file code logic is changed to increase the association logic between the service data collection and the service data and the performance data, so that the generated enhanced log contains the service data and the association relationship between the service data and the performance data.
6. The method for realizing global service monitoring based on log enhancement as claimed in claim 1, wherein said step S4 further comprises the steps of after the log platform performs data cleaning on the collected log data according to a regular expression cutting template, performing log data Search and storage through API provided by Elastic Search; and according to the service types, the query sentences of each service scene are obtained through sorting, and a service data query unit for querying through log keywords, reports and user behaviors is formed.
7. The method for implementing global service monitoring based on log enhancement as claimed in claim 6, wherein the presentation page of the log platform in step S5 is provided with a query entry, and the query entry invokes a service data query unit to perform service performance status query via an Elastic Search engine.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110293088.0A CN112988516B (en) | 2021-03-18 | Method for realizing global service monitoring based on log enhancement |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110293088.0A CN112988516B (en) | 2021-03-18 | Method for realizing global service monitoring based on log enhancement |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112988516A true CN112988516A (en) | 2021-06-18 |
| CN112988516B CN112988516B (en) | 2024-06-21 |
Family
ID=
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114338443A (en) * | 2021-12-20 | 2022-04-12 | 江苏云工场信息技术有限公司 | CDN node full link monitoring method and device |
| CN115801353A (en) * | 2022-11-03 | 2023-03-14 | 智网安云(武汉)信息技术有限公司 | Linkage script processing method after real-time aggregation of safety event logs based on big data level |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010206638A (en) * | 2009-03-04 | 2010-09-16 | Toshiba Corp | Monitoring device and method of storing business log information |
| CN107832187A (en) * | 2017-10-18 | 2018-03-23 | 广西电网有限责任公司电力科学研究院 | A kind of power transmission and transformation equipment state monitoring system |
| CN108874614A (en) * | 2017-05-11 | 2018-11-23 | 上海宏时数据系统有限公司 | A kind of big data log intelligent analysis system and method |
| CN110083436A (en) * | 2019-05-14 | 2019-08-02 | 上海理想信息产业（集团）有限公司 | A kind of business datum real-time monitoring system and method based on Java bytecode enhancing technology |
| CN110704277A (en) * | 2019-09-27 | 2020-01-17 | 中电万维信息技术有限责任公司 | Method for monitoring application performance, related equipment and storage medium |
| CN111026601A (en) * | 2019-09-23 | 2020-04-17 | 拉扎斯网络科技（上海）有限公司 | Monitoring method and device for Java application system, electronic equipment and storage medium |
| CN112035317A (en) * | 2020-08-28 | 2020-12-04 | 北京浪潮数据技术有限公司 | Micro-service link monitoring method, device, equipment and medium |
| CN112118153A (en) * | 2020-09-06 | 2020-12-22 | 苏州浪潮智能科技有限公司 | Grpc and spring mvc-based link monitoring method and system |
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2010206638A (en) * | 2009-03-04 | 2010-09-16 | Toshiba Corp | Monitoring device and method of storing business log information |
| CN108874614A (en) * | 2017-05-11 | 2018-11-23 | 上海宏时数据系统有限公司 | A kind of big data log intelligent analysis system and method |
| CN107832187A (en) * | 2017-10-18 | 2018-03-23 | 广西电网有限责任公司电力科学研究院 | A kind of power transmission and transformation equipment state monitoring system |
| CN110083436A (en) * | 2019-05-14 | 2019-08-02 | 上海理想信息产业（集团）有限公司 | A kind of business datum real-time monitoring system and method based on Java bytecode enhancing technology |
| CN111026601A (en) * | 2019-09-23 | 2020-04-17 | 拉扎斯网络科技（上海）有限公司 | Monitoring method and device for Java application system, electronic equipment and storage medium |
| CN110704277A (en) * | 2019-09-27 | 2020-01-17 | 中电万维信息技术有限责任公司 | Method for monitoring application performance, related equipment and storage medium |
| CN112035317A (en) * | 2020-08-28 | 2020-12-04 | 北京浪潮数据技术有限公司 | Micro-service link monitoring method, device, equipment and medium |
| CN112118153A (en) * | 2020-09-06 | 2020-12-22 | 苏州浪潮智能科技有限公司 | Grpc and spring mvc-based link monitoring method and system |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114338443A (en) * | 2021-12-20 | 2022-04-12 | 江苏云工场信息技术有限公司 | CDN node full link monitoring method and device |
| CN114338443B (en) * | 2021-12-20 | 2022-07-19 | 江苏云工场信息技术有限公司 | CDN node full link monitoring method and device |
| CN115801353A (en) * | 2022-11-03 | 2023-03-14 | 智网安云(武汉)信息技术有限公司 | Linkage script processing method after real-time aggregation of safety event logs based on big data level |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ratanaworabhan et al. | {JSMeter}: Comparing the Behavior of {JavaScript} Benchmarks with Real Web Applications | |
| Agarwal et al. | Making contextual decisions with low technical debt | |
| Kagdi et al. | Blending conceptual and evolutionary couplings to support change impact analysis in source code | |
| US9703675B2 (en) | Structured logging and instrumentation framework | |
| CN109902105B (en) | Data query system, method, device and storage medium for micro-service architecture | |
| Ali et al. | Trustrace: Mining software repositories to improve the accuracy of requirement traceability links | |
| US8352445B2 (en) | Development environment integration with version history tools | |
| US8209665B2 (en) | Identification of topics in source code | |
| US8019756B2 (en) | Computer apparatus, computer program and method, for calculating importance of electronic document on computer network, based on comments on electronic document included in another electronic document associated with former electronic document | |
| CN112486820B (en) | Method, apparatus, device and storage medium for testing code | |
| Mateos et al. | Detecting WSDL bad practices in code–first Web Services | |
| CN110825618B (en) | Method and related device for generating test case | |
| US10210211B2 (en) | Code searching and ranking | |
| WO2012148293A1 (en) | Using feedback reports to determine performance of an application in a geographic location | |
| US9852217B2 (en) | Searching and ranking of code in videos | |
| US8793653B2 (en) | Program code library selection in an integrated development environment | |
| WO2015077261A1 (en) | Validating software characteristics | |
| KR102593171B1 (en) | Information processing method and device, electronic equipment and storage medium | |
| CN105550206B (en) | The edition control method and device of structured query sentence | |
| US20090089119A1 (en) | Method, Apparatus, and Software System for Providing Personalized Support to Customer | |
| Gao et al. | Aibench scenario: Scenario-distilling ai benchmarking | |
| CN107391528B (en) | Front-end component dependent information searching method and equipment | |
| CN110351131B (en) | Monitoring method and device for distributed link and electronic equipment | |
| CN112084150A (en) | Model training method, data retrieval method, device, equipment and storage medium | |
| US20220152474A1 (en) | Developing implicit metadata for data stores |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |