CN110413496B - Method for realizing componentized collection of electronic license operation data - Google Patents
Method for realizing componentized collection of electronic license operation data Download PDFInfo
- Publication number
- CN110413496B CN110413496B CN201910686104.5A CN201910686104A CN110413496B CN 110413496 B CN110413496 B CN 110413496B CN 201910686104 A CN201910686104 A CN 201910686104A CN 110413496 B CN110413496 B CN 110413496B
- Authority
- CN
- China
- Prior art keywords
- log
- data
- module
- analysis system
- micro
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a method for realizing the modular collection of electronic license operation data, which provides an operation data collection platform; the system comprises a log tool module, a micro-service plug-in module, an analysis system and a data storage module; the log tool module and the micro-service plug-in module collect data and send the data collected by the log tool module and the micro-service plug-in module to the analysis system; the analysis system sends the received data to the data storage module for data storage; the analysis system can also acquire the data of the data storage module in a manual retrieval or timing task mode for displaying or reporting statistics. The invention is not limited by machine account, only a few people have authority for safety of the machine in the production environment, but the analysis system of the invention can conveniently distribute account and authority to people and does not affect the safety of the host in the production environment. The data can be effectively counted and analyzed; the utilization rate of data is improved, and a uniform retrieval space is provided for retrieval convenience.
Description
Technical Field
The invention relates to the field of electronic license, data collection and modularization, in particular to a method for realizing the modularization collection of electronic license operation data.
Background
The operation data mainly comprises a system log, an application program log and a safety log. And system operation and development personnel can know the information of software and hardware of the server and check errors in the configuration process and the reasons for the errors through logs. The load and the performance safety of the server can be known by frequently analyzing the logs, so that measures can be taken in time to correct errors. But the current situation is that: the electronic license platform has a plurality of applications, and some of the applications have collected operation data and some of the applications have not. And the way of collection and the way of storage are different. In the technical aspect, system.out.println, logback, log4j and operation data are directly stored on mysql or mongoDB.
Typically, the operational data is distributed across different devices stored. If tens or hundreds of servers are to be managed, the operational data is also reviewed using the conventional method of logging into each machine in turn. This can be cumbersome and inefficient. Centralized operational data management is used.
After the operation data is managed in a centralized manner, statistics and retrieval of the operation data become a relatively troublesome matter, and generally, retrieval and statistics can be realized by using Linux commands such as grep, awk and wc, but the method is still used for requirements such as higher query, sequencing and statistics and huge machine number. The log information on all the machines is collected and gathered together by carrying out centralized management on the log. The complete log data has a very important role:
1) information search: and searching log information, positioning the corresponding bug and finding out a solution.
2) Service diagnosis: by carrying out statistics and analysis on the log information, the load and the service running state of the server are known, and a time-consuming request is found for optimization and the like.
3) And (3) data analysis: if the log is formatted, further data analysis can be performed to count and aggregate meaningful information, such as finding out the service of interest to the TOP10 user according to the commodity id in the request.
4) A chain analysis is invoked. The collecting system also collects important information such as user identification, request module and the like, and information such as an initiating address and the like. Through it, can track whole calling chain, also be convenient for investigate the problem.
Disclosure of Invention
In view of the above, the present invention provides a method for implementing the modular collection of electronic license operation data, which can uniformly analyze and retrieve the log and improve the collection and utilization process of the operation data.
The invention is realized by adopting the following scheme: a method for realizing the modularization collection of the running data of electronic license provides a running data collection platform; the running data collection platform comprises a log tool module, a micro-service plug-in module, a log analysis system and a data storage module; the log tool module and the micro service plug-in module collect data and send the data collected by the log tool module and the micro service plug-in module to the log analysis system; the log analysis system sends the received data to the data storage module for data storage; the log analysis system can also acquire the data of the data storage module in a manual retrieval or timing task mode for displaying or counting reports;
after the log analysis system stores the data in the storage module, a plurality of timing tasks are built in the log analysis system. These timed tasks will generate some simple reports by date or other dimension based on the data in the data module. Meanwhile, the log analysis system also provides a visual retrieval interface, and a user can input retrieval conditions such as keywords and time periods on the interface to search data (logs) desired by the user.
The platform carries out electronic license operation data componentization collection according to the following steps:
step S1: carrying out self-defined logs: the log tool module obtains configuration from a configuration center; judging which log engine is from the flag bit in the configuration; if the engine is a local engine, logging is carried out by using a simple log interface; if the unified engine is the unified engine, a log tool module is used, log content, log recording time and code line information of the log are sent to the log analysis system in a json format, and step S7 is executed;
step S2: and (3) statistical log storage: the developer calls an interface of the log tool module to perform statistical data entry, and executes the step S7;
step S3: carrying out micro-service calling: if the calling party uses the log plug-in and does not shield the log function, the log function is started; judging whether a calling party starts a log or not; if the log is started, the caller starts logging and executes step S4; otherwise, executing step S5;
step S4: the logging tool module generates a unique identifier for the current context for invoking a chain tracking function; if the micro-service call is initiated, the identifier is transmitted to the callee; by this mechanism, the logging tool module identifies whether an operation was originally initiated or invoked by a third party; if yes, the caller starts logging and executes step S6; otherwise, directly executing the step S6;
step S5: the called party starts logging, acquires the additional log content transferred by the client and finally executes the step S6;
step S6: recording the initiation position of the micro service, the accessed interface, the access parameter, the return value, the success or failure, the abnormal information and the time spent on processing, sending the initiation position of the micro service, the accessed interface, the access parameter and the result to a log service system, and executing step S7;
step S7: supplementing an application identifier, a local ip of a log recorder, a port monitored by the log recorder, a context identifier, a field allowing user customization and the like serving as customized content into a log, and then sending the log to the log analysis system in a batch mode; the log analysis system parses the type of each log from the batch and then stores it into a suite of big data search and real-time analysis engines.
Further, the parsed log types described in step S7 include, but are not limited to, custom logs, statistical logs, and microservice logs.
Compared with the prior art, the invention has the following beneficial effects:
the invention is not limited by machine account, only a few people have authority for safety of the machine in the production environment, but the log analysis system of the invention can conveniently distribute account and authority to people without influencing the safety of the host in the production environment. The data can be effectively counted and analyzed; the utilization rate of data is improved, the retrieval is convenient, and a uniform retrieval space is provided.
Drawings
FIG. 1 is a flow chart of an embodiment of the present invention.
Detailed Description
The invention is further explained by the following embodiments in conjunction with the drawings.
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
As shown in fig. 1, the present embodiment provides a method for implementing componentized collection of electronic license operation data,
providing an operation data collection platform; the running data collection platform comprises a log tool module, a micro-service plug-in module, a log analysis system and a data storage module; the log tool module and the micro-service plug-in module collect data and send the data collected by the log tool module and the micro-service plug-in module to the log analysis system; the log analysis system sends the received data to the data storage module for data storage; the log analysis system can also acquire the data of the data storage module in a manual retrieval or timing task mode for displaying or counting reports;
the platform carries out electronic license operation data componentization collection according to the following steps:
step S1: carrying out self-defined logs: the log tool module obtains configuration from a configuration center; judging which log engine is from the flag bit in the configuration; if the engine is a local engine, logging is carried out by using a simple log interface; if the unified engine is used, a log tool module is used, log content, log recording time and code line information of the log are sent to the log analysis system in a json format, and step 7 is executed;
step S2: and (3) statistical log storage: the developer calls an interface of the log tool module to perform statistical data entry, and executes the step S7;
step S3: carrying out micro-service calling: if the calling party uses the log plug-in and does not shield the log function, the log function is started; judging whether a calling party starts a log or not; if the log is enabled, the caller starts logging and executes step S4; otherwise, executing step S5;
step S4: the logging tool module generates a unique identifier for the current context for invoking a chain tracking function; if the micro-service call is initiated, the identifier is transmitted to the callee; by this mechanism, the logging tool module identifies whether an operation was originally initiated or invoked by a third party (identifier); if yes, the caller starts logging and executes step S6; otherwise, directly executing step S6;
step S5: the called party starts logging, acquires the additional log content transferred by the client and finally executes the step S6;
step S6: recording the initiation position, the accessed interface, the access parameter, the return value, the success or failure, the abnormal information and the time spent on processing of the micro service, sending the initiation position, the accessed interface, the access parameter and the result of the micro service to a log service system, and executing the step S7;
step S7: supplementing an application identifier, a local ip of a log recorder, a port monitored by the log recorder, a context identifier, a field allowing user customization and the like into a log as customized content, and then sending the log to the log analysis system in a batch mode; the log analysis system parses the type of each log from the batch and then stores it to the big data search and real-time analysis engine suite.
In the present embodiment, the types of the parsed logs described in step S7 include, but are not limited to, custom logs, statistical logs, and microservice logs.
1) The log tool is a manual operation data collection tool, and can collect and filter logs. It can write a log of fixed contents, and can also write data of variable data items. The variable data items include statistical data and the like, and are mainly used for statistical analysis.
2) The microservice plug-in is a code-hack-free plug-in for dubbo, and if the developer does not have custom log content, no additional code development is required. The microservice plug-in can automatically collect interface requests, exception logs, request time consumption, launch location, etc. at the requesting or called end. The log is collected at the request side by default, and if the micro-service plug-in is not installed or disabled at the request side, the log is collected at the service side. And moreover, the method also supports the transmission of identifiers such as user id and the like on the whole calling chain, so that the data quality is improved, and the eagle eye tracking function is also realized.
3) The log analysis system automatically maintains an index repository on the elastic search according to the configuration. When receiving the data, the data is classified and then stored in the index library corresponding to the elastic search. And provides retrieval and analysis functions.
4) The data storage uses an elastic search, which is an open source distributed search service system based on Lucene. It is characterized in that: distributed, zero configuration, automatic discovery, index automatic fragmentation, index copy mechanism, restful style interface, multiple data sources, automatic search load, and the like. It provides a distributed multi-user capable full-text search engine based on RESTful web interface. The elastic search is developed in Java and published as open source under the Apache licensing terms, and is the second popular enterprise search engine. The design is used in cloud computing, can achieve real-time search, and is stable, reliable, quick, and convenient to install and use. In the elastic search, data of all nodes are equal.
Preferably, the embodiment divides the operation data collection into manual data collection and micro-service data automatic collection. Whatever data is, some additional information, such as the current system ip address, the user identification (if any) associated with the data, etc., is added before sending to the log analysis system. Then, the data are uniformly stored in an elastic search, and then the logs can be uniformly analyzed and retrieved. The collection and utilization processes of the operation data are greatly improved.
The above description is only a preferred embodiment of the present invention, and all equivalent changes and modifications made in accordance with the claims of the present invention should be covered by the present invention.
Claims (2)
1. A method for realizing the componentized collection of electronic license operation data is characterized in that: providing an operation data collection platform; the running data collection platform comprises a log tool module, a micro-service plug-in module, a log analysis system and a data storage module; the log tool module and the micro service plug-in module collect data and send the data collected by the log tool module and the micro service plug-in module to the log analysis system; the log analysis system sends the received data to the data storage module for data storage; the log analysis system can also acquire the data of the data storage module in a manual retrieval or timing task mode for displaying or report statistics;
the platform carries out electronic license operation data componentization collection according to the following steps:
step S1: carrying out self-defined logs: the log tool module obtains configuration from a configuration center; judging which log engine is from the flag bit in the configuration; if the local engine is the local engine, a general simple log interface is used for log recording; if the unified engine is used, a log tool module is used, log content, log recording time and code line information of the log are sent to the log analysis system in a json format, and step 7 is executed;
step S2: and (3) statistical log storage: the developer calls an interface of the log tool module to perform statistical data entry, and executes the step S7;
step S3: carrying out micro-service calling: if the calling party uses the log plug-in and does not shield the log function, the log function is started; judging whether a calling party starts a log or not; if the log is started, the caller starts logging and executes step S4; otherwise, executing step S5;
step S4: the log tool module generates a unique identifier for the current context for calling a chain tracking function; if the micro-service call is initiated, the identifier is transmitted to the callee; by this mechanism, the logging tool module identifies whether an operation was originally initiated or invoked by a third party; if the calling party is called by the third party, starting the logging by the calling party and executing the step S6; otherwise, directly executing step S6;
step S5: the called party starts logging, acquires the additional log content transferred by the client and finally executes the step S6;
step S6: recording the initiation position of the micro service, the accessed interface, the access parameter, the return value, the success or failure, the abnormal information and the time spent on processing, sending the initiation position of the micro service, the accessed interface, the access parameter and the result to a log service system, and executing step S7;
step S7: the application identification, the local ip of the log recorder, the port monitored by the log recorder, the context identification and the field allowing the user to define are taken as the defined content to be supplemented into the log, and then the log is sent to a log analysis system in a batch mode; the log analysis system parses the type of each log from the batch and then stores the logs into a suite of big data search and real-time analysis engines.
2. The method for realizing the componentized collection of the electronic license operation data according to claim 1, wherein the method comprises the following steps: the log types analyzed in step S7 include a custom log, a statistical log, and a microservice log.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910686104.5A CN110413496B (en) | 2019-07-29 | 2019-07-29 | Method for realizing componentized collection of electronic license operation data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910686104.5A CN110413496B (en) | 2019-07-29 | 2019-07-29 | Method for realizing componentized collection of electronic license operation data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110413496A CN110413496A (en) | 2019-11-05 |
| CN110413496B true CN110413496B (en) | 2022-08-19 |
Family
ID=68363640
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910686104.5A Active CN110413496B (en) | 2019-07-29 | 2019-07-29 | Method for realizing componentized collection of electronic license operation data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110413496B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112364189B (en) * | 2020-11-16 | 2023-02-21 | 浪潮云信息技术股份公司 | Electronic certificate application method based on ES service |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9626328B1 (en) * | 2013-06-19 | 2017-04-18 | EMC IP Holding Company LLC | Method and system for on-demand aggregated logging for distributed systems |
| WO2017166644A1 (en) * | 2016-03-31 | 2017-10-05 | 乐视控股（北京）有限公司 | Data acquisition method and system |
| CN109756364A (en) * | 2018-12-07 | 2019-05-14 | 成都四方伟业软件股份有限公司 | A kind of micro services performance optimization system and analysis method based on log analysis |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA2419305C (en) * | 2003-02-20 | 2006-03-21 | Ibm Canada Limited - Ibm Canada Limitee | Unified logging service for distributed applications |
| CN101192227B (en) * | 2006-11-30 | 2011-05-25 | 阿里巴巴集团控股有限公司 | Log file analytical method and system based on distributed type computing network |
| CN103684898B (en) * | 2012-09-14 | 2017-06-23 | 阿里巴巴集团控股有限公司 | It is a kind of to monitor the method and device that user's request is run in a distributed system |
| US9729671B2 (en) * | 2014-10-05 | 2017-08-08 | YScope Inc. | Systems and processes for computer log analysis |
-
2019
- 2019-07-29 CN CN201910686104.5A patent/CN110413496B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9626328B1 (en) * | 2013-06-19 | 2017-04-18 | EMC IP Holding Company LLC | Method and system for on-demand aggregated logging for distributed systems |
| WO2017166644A1 (en) * | 2016-03-31 | 2017-10-05 | 乐视控股（北京）有限公司 | Data acquisition method and system |
| CN109756364A (en) * | 2018-12-07 | 2019-05-14 | 成都四方伟业软件股份有限公司 | A kind of micro services performance optimization system and analysis method based on log analysis |
Non-Patent Citations (1)
| Title |
|---|
| 基于微服务架构的日志系统;黄媛媛等;《电子技术与软件工程》;20170115(第02期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110413496A (en) | 2019-11-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11379475B2 (en) | Analyzing tags associated with high-latency and error spans for instrumented software | |
| US11775343B1 (en) | Duty cycle estimation for job assignment | |
| US11409645B1 (en) | Intermittent failure metrics in technological processes | |
| US11775501B2 (en) | Trace and span sampling and analysis for instrumented software | |
| KR102614428B1 (en) | Systems and methods for updating multi-tier cloud-based application stacks | |
| US7913233B2 (en) | Performance analyzer | |
| US11366713B2 (en) | System and method for automatically identifying and resolving computing errors | |
| US20180143897A1 (en) | Determining idle testing periods | |
| US11301475B1 (en) | Transmission handling of analytics query response | |
| US20200319947A1 (en) | Api and streaming solution for documenting data lineage | |
| CN110851324A (en) | Log-based routing inspection processing method and device, electronic equipment and storage medium | |
| CN112306700A (en) | Abnormal RPC request diagnosis method and device | |
| CN113760677A (en) | Abnormal link analysis method, device, equipment and storage medium | |
| CN112231180A (en) | SQL monitoring method and device based on cloud environment | |
| CN110413496B (en) | Method for realizing componentized collection of electronic license operation data | |
| CN113407511A (en) | Log aggregation method, log aggregation equipment and computer program product | |
| CN117271584A (en) | Data processing method and device, computer readable storage medium and electronic equipment | |
| CN112559525A (en) | Data checking system, method, device and server | |
| CN112187509A (en) | Multi-architecture cloud platform execution log management method, system, terminal and storage medium | |
| CN111368039B (en) | Data management system | |
| US20190190981A1 (en) | Intelligent trace generation from compact transaction runtime data | |
| CN115757041B (en) | Method for collecting dynamically configurable multi-cluster logs and application | |
| US11755453B1 (en) | Performing iterative entity discovery and instrumentation | |
| CN117389841B (en) | Method and device for monitoring accelerator resources, cluster equipment and storage medium | |
| CN116976984A (en) | Advertisement report acquisition method and device based on MetaAPI and related medium thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |