CN111162932A - API gateway monitoring method based on log analysis - Google Patents
API gateway monitoring method based on log analysis Download PDFInfo
- Publication number
- CN111162932A CN111162932A CN201911273857.XA CN201911273857A CN111162932A CN 111162932 A CN111162932 A CN 111162932A CN 201911273857 A CN201911273857 A CN 201911273857A CN 111162932 A CN111162932 A CN 111162932A
- Authority
- CN
- China
- Prior art keywords
- gateway
- api gateway
- log
- monitoring
- method based
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0246—Exchanging or transporting network management information using the Internet; Embedding network management web servers in network elements; Web-services-based protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
- H04L43/045—Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0817—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Environmental & Geological Engineering (AREA)
- Debugging And Monitoring (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses an API gateway monitoring method based on log analysis, which is characterized in that: compiling a collecting program for recording call logs, defining the collecting program as a Bo-log plug-in, installing the collecting program to the API gateway in a plug-in mode, generating logs for each call of the API gateway by utilizing the Bo-log plug-in, sending the logs to flow type calculation, obtaining required monitoring data by analyzing log sorting and calculation, and displaying monitoring performance, configuring gateway functions and governing strategies through gateway management service. By applying the monitoring solution of the invention, the collected logs are sorted and analyzed by the stream-oriented computation deployed outside the API gateway, thereby eliminating the performance consumption of the API gateway and ensuring the performance of the routing service; meanwhile, the monitoring data precision is improved, so that the response of the back-end service is facilitated.
Description
Technical Field
The invention relates to the technical field of software of computers and network products, in particular to an API gateway monitoring method based on log analysis.
Background
With the rapid development of IT technology, the Internet application gradually becomes an important driving force for business innovation and enterprise quality improvement and efficiency improvement. Therefore, the service quality of the internet application is guaranteed to be very important, and the operation and maintenance monitoring becomes the important factor in the operation and the service operation and maintenance of the platform. The current monitoring software does not exceed a six-layer monitoring system: 1. infrastructure monitoring, i.e., servers, switches, etc.; 2. system layer monitoring, operating system, CPU, memory, network, etc.; 3. monitoring a database and middleware; 4. an application layer, application performance monitoring, service response time, QPS, and the like; 5. service monitoring, registration login, order payment and the like; 6. end user experience monitoring, i.e. browser monitoring, user perception monitoring, etc.
The six-layer monitoring system strictly monitors the whole network communication theoretically, and covers each communication link from the client to the server. But the monitoring requirements of most external open platforms, which provide enterprise or platform functional service websites, cannot be met, and the actual use interface is provided by the bottom enterprise gateway. For development platforms or service middleboxes of most enterprises, metering charging, health degree, calling performance and the like of a client side to API use cannot be effectively and intuitively monitored by a traditional six-layer monitoring system through calling performance and calling statistics of a gateway.
The API gateway is the only entry for external access to the internal programs of the enterprise, and is a performance bottleneck for all external services in the face of a large number of interface forwarding, authentication and authorization. Monitoring of the API gateway is also an essential function in the service chain, and monitoring each call and gathering numerous information content will also increase performance consumption at the gateway.
Disclosure of Invention
The invention aims to provide an API gateway monitoring method based on log analysis, which aims at providing data statistics and accounting for an API calling process, calling metering and API performance at a gateway, mastering the calling performance, avoiding problems and preventing avalanche.
The technical solution for achieving the above object of the present invention is an API gateway monitoring method based on log analysis, which is characterized in that: compiling a collecting program for recording call logs, defining the collecting program as a Bo-log plug-in, installing the collecting program to the API gateway in a plug-in mode, generating logs for each call of the API gateway by utilizing the Bo-log plug-in, sending the logs to flow type calculation, obtaining required monitoring data by analyzing log sorting and calculation, and displaying monitoring performance, configuring gateway functions and governing strategies through gateway management service.
In the API gateway monitoring method based on log analysis, the description information of the log further includes caller information, gateway routing information, gateway service information, backend instance information, call time, end time, return result, and usage flow.
In the API gateway monitoring method based on log analysis, the log is in a JSON format.
According to the API gateway monitoring method based on log analysis, furthermore, the stream type calculation is deployed outside the API gateway.
In the API gateway monitoring method based on log analysis, further, the API gateway is deployed with a bottom storage service and a front-end gateway management service, the monitoring data obtained by stream computing is stored in a background Elastic Search memory, and the gateway management service acquires the monitoring data from the Elastic Search memory and displays the monitoring data based on a Web page.
In the above API gateway monitoring method based on log analysis, further, the monitoring data at least includes call volume statistics, call performance statistics, and usage flow statistics.
In the API gateway monitoring method based on log analysis, furthermore, the API gateway is also deployed with a back-end service at least including an alarm, a statistical form, performance-based fusing, current limiting, and scaling.
The API gateway monitoring method has the prominent substantive characteristics and remarkable progress: the collected logs are sorted and analyzed through the stream type calculation deployed outside the API gateway, so that the performance consumption of the API gateway is eliminated, and the performance of the routing service is guaranteed; meanwhile, the monitoring data precision is improved, so that the response of the back-end service is facilitated.
Drawings
FIG. 1 is a block diagram of the overall architecture of the API gateway implementing the functional services of the present invention.
Fig. 2 is a schematic block diagram of the API gateway implementing the monitoring function of the present invention.
Detailed Description
The following detailed description of the embodiments of the present invention is provided in connection with the accompanying drawings for the purpose of understanding and controlling the technical solutions of the present invention, so as to define the protection scope of the present invention more clearly.
Before the API gateway monitoring method of the present invention is further introduced, the architecture of the API gateway function level is simply introduced. The main functions of the API gateway are the route forwarding of the interface and the authentication of the access, the back end of the gateway is the bottom layer service or application, and the interface is exposed at the API gateway for external calling. As shown in fig. 1, the API gateway is typically deployed as multiple microservice processes, including underlying gateway routing services, streaming computing services, underlying storage services, front-end management web services, and portions of the required extensibility services (alarms, logs, etc.).
The underlying gateway routing service is a service module that provides the primary gateway function, which many open source components can also provide. The system is mainly responsible for routing forwarding, a background inputs a service address and an access mode into the program, and the program configures routing and provides services for the outside. The service is a core component of the API gateway and mainly bears external service call.
The stream type computing service sets computing rules, and has the main functions of analyzing and sorting call logs, computing a specific value required by a platform, and then caching a processing result or forwarding data to other services for use.
The gateway management service is mainly used for interaction with a manager, gateway function configuration, monitoring performance display, other administration strategies configuration and the like.
The present invention focuses only on the interfaces exposed at the API gateway, and the monitoring data focuses on the following aspects: for realizing call charging, the call quantity statistics, that is, the call quantity of a certain interface API, needs to obtain the information of the caller by using the user as a unit, so as to count the call quantity. And the calling performance statistics of the API gateway and the calling performance statistics are carried out, in a certain calling process, the calling starting time and the API gateway feedback time (namely the ending time) are recorded, the response time of the API calling at the current time is calculated, and the API is taken as the dimension to carry out statistics on the average response time of the API calling in a certain period of time. The flow usage statistics of the three is used, the total message size in API call is counted, the network flow usage is calculated, and the network flow usage can also be used as auxiliary reference information for metering and charging. Based on the calling monitoring, performance monitoring information such as QPS, flow throughput, availability in a certain time and the like is calculated in a statistical mode.
The main purpose of the invention is to record each call at the API gateway and record all information required by monitoring, so that the API gateway can be used for monitoring and can be convenient for tracing after. The summary technical means for realizing the method is as follows: writing a collecting program for recording call logs, defining the collecting program as a Bo-log plug-in, installing the collecting program to the API gateway in a plug-in mode, generating a detailed log for each call generated by the API gateway by using the Bo-log plug-in, sending the log to flow type calculation, obtaining required monitoring data by analyzing log arrangement and calculation, and displaying monitoring performance, configuring gateway functions and governing strategies through gateway management service.
As shown in fig. 2, a collector for logging calls is written in lua language, which can be understood or defined as a Bo-log plug-in to the API gateway, adding it to the gateway resources. The log plug-in is effective to each service and routing resource in the gateway, log information can be generated by each calling, and the log mainly records caller information, gateway routing information, gateway service information, rear-end instance information, calling time, ending time, return results, flow use conditions and the like.
And after each call, the log content is only transmitted to the background streaming calculation in the JSON format through the log information generated by the plug-in, and the data required by monitoring is acquired through the streaming calculation. The streaming computation is deployed outside the API gateway, and the monitoring data includes, but is not limited to, call volume statistics, call performance statistics, usage traffic statistics, and the like.
The API gateway is deployed with a bottom storage service and a front-end gateway management service, taking an Elastic Search memory as an example, monitoring data obtained by streaming computation is stored in a background Elastic Search memory, so as to provide basic data for many functional requirements in the API gateway. And facing to the manager, the gateway management service acquires the monitoring data from the Elastic Search memory and displays the monitoring data based on the Web page.
Of course, the underlying storage service described above may implement storage in other ways besides Elastic Search storage.
In addition, the API gateway is also provided with back-end services at least comprising alarms, statistical reports, performance-based fusing, current limiting, capacity expansion and the like. After the flow calculation is completed, corresponding functional operation is automatically triggered when corresponding monitoring data reaches a preset rule or threshold value.
In summary, the API gateway monitoring scheme introduction and the embodiment detailed description of the present invention can be seen, and the scheme has substantial features and progressiveness: the invention adopts the log collection program, is installed on the API gateway in a plug-in mode, and only sends the log in the JSON format to the stream type calculation for each call, thereby reducing the performance consumption to the lowest.
The program for really sorting and analyzing the logs is in the stream computing, and the stream computing is deployed outside the API gateway, so that the performance consumption of the API gateway is not increased. The independent stream type calculation program can bear all log analysis work, and the performance of the API gateway has no obvious change in actual test and use.
In addition, the monitoring data collected by the invention has extremely high accuracy, and the successful calling number and the failed calling number can be accurately collected, so that the accurate response effect is correspondingly achieved on error early warning, fusing current limiting and background service expansion and contraction.
In addition to the above embodiments, the present invention may have other embodiments, and any technical solutions formed by equivalent substitutions or equivalent transformations are within the scope of the present invention as claimed.
Claims (7)
1. An API gateway monitoring method based on log analysis is characterized in that: compiling a collecting program for recording call logs, defining the collecting program as a Bo-log plug-in, installing the collecting program to the API gateway in a plug-in mode, generating logs for each call of the API gateway by utilizing the Bo-log plug-in, sending the logs to flow type calculation, obtaining required monitoring data by analyzing log sorting and calculation, and displaying monitoring performance, configuring gateway functions and governing strategies through gateway management service.
2. The API gateway monitoring method based on log analysis of claim 1, wherein: the record information of the log comprises caller information, gateway routing information, gateway service information, back-end instance information, calling time, end time, return result and use flow.
3. The API gateway monitoring method based on log analysis of claim 1, wherein: the log is in a JSON format.
4. The API gateway monitoring method based on log analysis of claim 1, wherein: the streaming computation is deployed outside the API gateway.
5. The API gateway monitoring method based on log analysis of claim 1, wherein: the API gateway is deployed with a bottom storage service and a front-end gateway management service, the monitoring data obtained through stream computing are stored in a background Elastic Search memory, and the gateway management service acquires the monitoring data from the Elastic Search memory and displays the monitoring data on the basis of a Web page.
6. The API gateway monitoring method based on log analysis of claim 1 or 5, wherein: the monitoring data at least comprises calling quantity statistics, calling performance statistics and using flow statistics.
7. The API gateway monitoring method based on log analysis of claim 1 or 5, wherein: the API gateway is also provided with at least an alarm, a statistical form, performance-based fusing, current limiting and capacity expansion and contraction.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911273857.XA CN111162932A (en) | 2019-12-12 | 2019-12-12 | API gateway monitoring method based on log analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911273857.XA CN111162932A (en) | 2019-12-12 | 2019-12-12 | API gateway monitoring method based on log analysis |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111162932A true CN111162932A (en) | 2020-05-15 |
Family
ID=70556798
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911273857.XA Pending CN111162932A (en) | 2019-12-12 | 2019-12-12 | API gateway monitoring method based on log analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111162932A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112272115A (en) * | 2020-10-28 | 2021-01-26 | 首都信息发展股份有限公司 | API gateway based on stream processing and log collection method thereof |
| CN112527601A (en) * | 2020-12-17 | 2021-03-19 | 航天信息股份有限公司 | Monitoring early warning method and device |
| CN113259880A (en) * | 2021-05-20 | 2021-08-13 | 洛阳轴承研究所有限公司 | Micro-service safety charging method based on kong gateway |
| CN113726566A (en) * | 2021-08-25 | 2021-11-30 | 杭州数政科技有限公司 | Service gateway device |
| CN114938353A (en) * | 2022-05-27 | 2022-08-23 | 中国银行股份有限公司 | Asynchronous notification current limiting method and system based on stream computing |
| CN115766527A (en) * | 2022-12-23 | 2023-03-07 | 普元信息技术股份有限公司 | Business analysis system and method based on API gateway inlet and outlet flow under trusted environment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140366118A1 (en) * | 2013-06-05 | 2014-12-11 | Fortinet, Inc. | Cloud based logging service |
| CN105574205A (en) * | 2016-01-18 | 2016-05-11 | 国家电网公司 | Dynamic log analyzing system for distributed computing environment |
| CN105827608A (en) * | 2016-03-31 | 2016-08-03 | 微梦创科网络科技（中国）有限公司 | Distributed API service abnormal user identification analysis method and reverse agent service gateway |
| CN106294091A (en) * | 2016-08-11 | 2017-01-04 | 福建富士通信息软件有限公司 | A kind of without intrusive mood daily record interception method for analyzing performance and system |
| CN109688097A (en) * | 2018-09-07 | 2019-04-26 | 平安科技（深圳）有限公司 | Website protection method, website protective device, website safeguard and storage medium |
| CN109756364A (en) * | 2018-12-07 | 2019-05-14 | 成都四方伟业软件股份有限公司 | A kind of micro services performance optimization system and analysis method based on log analysis |
-
2019
- 2019-12-12 CN CN201911273857.XA patent/CN111162932A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140366118A1 (en) * | 2013-06-05 | 2014-12-11 | Fortinet, Inc. | Cloud based logging service |
| CN105574205A (en) * | 2016-01-18 | 2016-05-11 | 国家电网公司 | Dynamic log analyzing system for distributed computing environment |
| CN105827608A (en) * | 2016-03-31 | 2016-08-03 | 微梦创科网络科技（中国）有限公司 | Distributed API service abnormal user identification analysis method and reverse agent service gateway |
| CN106294091A (en) * | 2016-08-11 | 2017-01-04 | 福建富士通信息软件有限公司 | A kind of without intrusive mood daily record interception method for analyzing performance and system |
| CN109688097A (en) * | 2018-09-07 | 2019-04-26 | 平安科技（深圳）有限公司 | Website protection method, website protective device, website safeguard and storage medium |
| CN109756364A (en) * | 2018-12-07 | 2019-05-14 | 成都四方伟业软件股份有限公司 | A kind of micro services performance optimization system and analysis method based on log analysis |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112272115A (en) * | 2020-10-28 | 2021-01-26 | 首都信息发展股份有限公司 | API gateway based on stream processing and log collection method thereof |
| CN112272115B (en) * | 2020-10-28 | 2023-03-28 | 首都信息发展股份有限公司 | API gateway based on stream processing and log collection method thereof |
| CN112527601A (en) * | 2020-12-17 | 2021-03-19 | 航天信息股份有限公司 | Monitoring early warning method and device |
| CN113259880A (en) * | 2021-05-20 | 2021-08-13 | 洛阳轴承研究所有限公司 | Micro-service safety charging method based on kong gateway |
| CN113726566A (en) * | 2021-08-25 | 2021-11-30 | 杭州数政科技有限公司 | Service gateway device |
| CN113726566B (en) * | 2021-08-25 | 2024-03-26 | 杭州数政科技有限公司 | Service gateway device |
| CN114938353A (en) * | 2022-05-27 | 2022-08-23 | 中国银行股份有限公司 | Asynchronous notification current limiting method and system based on stream computing |
| CN114938353B (en) * | 2022-05-27 | 2024-04-16 | 中国银行股份有限公司 | Asynchronous notification current limiting method and system based on stream computing |
| CN115766527A (en) * | 2022-12-23 | 2023-03-07 | 普元信息技术股份有限公司 | Business analysis system and method based on API gateway inlet and outlet flow under trusted environment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111162932A (en) | API gateway monitoring method based on log analysis | |
| CN107992398B (en) | Monitoring method and monitoring system of service system | |
| EP3379419B1 (en) | Situation analysis | |
| US11392416B2 (en) | Automated reconfiguration of real time data stream processing | |
| US9432462B2 (en) | Distributed metering and monitoring system | |
| CN107040863B (en) | Real-time service recommendation method and system | |
| US7356590B2 (en) | Distributed capture and aggregation of dynamic application usage information | |
| US8578017B2 (en) | Automatic correlation of service level agreement and operating level agreement | |
| US20170316321A1 (en) | Predictive rollup and caching for application performance data | |
| US20060026467A1 (en) | Method and apparatus for automatically discovering of application errors as a predictive metric for the functional health of enterprise applications | |
| US10452463B2 (en) | Predictive analytics on database wait events | |
| CN103023693B (en) | A kind of user behaviors log data management system and method | |
| US20180123922A1 (en) | Correlating performance outliers and network performance impacting event metric | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| CN104104734A (en) | Log analysis method and device | |
| US10439899B2 (en) | Service summary view | |
| US20190342189A1 (en) | Real-time ranking of monitored entities | |
| US8291061B2 (en) | Method and system for business-oriented web services management | |
| CN106559498A (en) | Air control data collection platform and its collection method | |
| CN112141832A (en) | Visual operation platform of elevator thing networking | |
| CN110266555B (en) | Method for analyzing website service request | |
| US10644971B2 (en) | Graph search in structured query language style query | |
| US20180121329A1 (en) | Uninstrumented code discovery | |
| CN114598622B (en) | Data monitoring method and device, storage medium and computer equipment | |
| EP3764232B1 (en) | Business transactions impact analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 215213 South Floor 4, Fuhua Building, No. 199 Chongwen Road, Suzhou Industrial Park, Jiangsu Province Applicant after: Jiangsu Boyun Technology Co.,Ltd. Address before: 215213 South Floor 4, Fuhua Building, No. 199 Chongwen Road, Suzhou Industrial Park, Jiangsu Province Applicant before: SUZHOU BONA XUNDONG SOFTWARE Co.,Ltd. |
|
| CB02 | Change of applicant information |