CN115333966A

CN115333966A - Nginx log analysis method, system and equipment based on topology

Info

Publication number: CN115333966A
Application number: CN202210963046.8A
Authority: CN
Inventors: 田标; 崔伟; 邓捷; 袁科; 易景平
Original assignee: Tianyi Digital Life Technology Co Ltd
Current assignee: Tianyi Digital Life Technology Co Ltd
Priority date: 2022-08-11
Filing date: 2022-08-11
Publication date: 2022-11-11
Anticipated expiration: 2042-08-11
Also published as: CN115333966B

Abstract

The invention relates to the technical field of computer network management, and discloses a Nginx log analysis method, system and equipment based on topology. The server where each Nginx instance is located obtains Nginx access records based on a Logstash configuration file or a pre-developed acquisition tool and sends the Nginx access records to a Kafka cluster for storage; the server side extracts a corresponding Nginx access record set from the Kafka cluster according to analysis requirements, analyzes the Nginx access record set to obtain the association relation between the user and each service system, calculates the access quality data between each association node according to corresponding operation quality data to further construct a topological graph, takes the flow direction of the request data as the direction of each edge of the graph, and sets the attribute of the corresponding edge according to the access quality data between the association nodes. The invention solves the technical problem of how to construct a topological graph reflecting the change conditions of the calling relationship and the running quality between users and related systems of the service global situation based on the Nginx log.

Description

Nginx log analysis method, system and equipment based on topology

Technical Field

The invention relates to the technical field of computer network management, in particular to a method, a system and equipment for analyzing Nginx logs based on topology.

Background

With the development of network technology, modern service systems become more and more complex, more and more servers and terminal devices are involved, and network topology is especially important in large-scale network management. From the overall view of the service, the operation quality data (such as performance, error, access amount and the like) and the change conditions thereof in the processes of user access and inter-system call can provide quantitative data support for timely discovery, deep analysis, post playback, verification and the like of the network security problems of the distributed system if the operation quality data and the change conditions thereof can be displayed in the form of a topological graph, so that the service guarantee work of the distributed system is greatly facilitated.

At present, the connection relationship between servers is usually analyzed according to logs such as connection and traffic between servers, and the network topology state between servers is determined according to the analyzed connection relationship. For example, patent application No. CN202111227745.8 provides a log and graph-based dynamic network topology graph generation method, system, processing device, and storage medium, which resolves access server IP, accessed server IP, connection time, user MAC address, connection mode, and traffic data by collecting server traffic, ssh, ftp, and web system logs, and further creates a directed graph structure using the resolved data, so that relevant persons can query the network topology state according to the created graph. However, the existing method can only process two-layer call relations from the source server to the target server, and cannot reflect the call relations and the operation quality change conditions between users of the service global and related systems, and the connection relations between the servers need to be analyzed by analyzing various log data, so that the practicability of the method is reduced.

Because the characteristics of high-performance reverse proxy of a plurality of backend systems with different protocols, convenient expansion and the like are widely applied, the log of the Nginx can record information such as source IP accessed by each user, time consumption of requests forwarded to each backend instance, error and the like. By analyzing the Nginx log, the calling relationship between the users of the service global and the related systems can be known. However, due to factors such as large data volume of the Nginx log, insufficient calling identification between different Nginx instances, and differences between different platforms, how to construct a topological graph reflecting the calling relationship and the running quality change condition between users and related systems of the service global based on the Nginx log is still lacking in the prior art.

Disclosure of Invention

The invention provides a Nginx log analysis method, a system and equipment based on topology, which solve the technical problem of how to construct a topological graph reflecting the change conditions of the call relationship and the operation quality between users and related systems of the service global situation based on Nginx logs.

The invention provides a topology-based Nginx log analysis method in a first aspect, which comprises the following steps:

determining Nginx log analysis requirements, wherein the Nginx log analysis requirements comprise a unique identifier of a target service and information of an analysis time period;

acquiring an Nginx access record set corresponding to the analysis time period from the Kafka cluster according to the unique identifier of the target service; a plurality of Nginx access records are stored in the Kafka cluster, each Nginx access record is obtained by acquiring and analyzing an Nginx access log through a server where a corresponding Nginx instance is located based on a preset Logstash configuration file or a pre-developed acquisition tool, the Nginx access record comprises call relation topological data and corresponding operation quality data from an access initiator to an access target, the access initiator is a user end, the server where the Nginx instance is located or a reverse proxy back-end system instance, and the access target is the server where the Nginx instance is located or the reverse proxy back-end system instance;

analyzing the calling relationship topological data in the Nginx access record set to obtain an association relationship among each user side, a server where each Nginx instance is located and each reverse proxy back-end system instance, and calculating a preset access quality index among association nodes according to the operation quality data in the Nginx access record set to obtain corresponding access quality data;

and constructing a corresponding topological graph according to the association relation and the access quality data, wherein the flow direction of the request data is taken as the direction of each edge of the topological graph, and the attribute of the corresponding edge is set according to the access quality data among the associated nodes.

According to an implementation manner of the first aspect of the present invention, the acquiring, from the Kafka cluster, the Nginx access record set corresponding to the analysis time period according to the unique identifier of the target service includes:

reading the Nginx access record set by using Logstash.

According to an implementation manner of the first aspect of the present invention, the analyzing each invocation relationship topology data in the nginnx access record set includes:

determining the addresses of all Nginx access records in the Nginx access record set from an access initiator to an access target;

and matching the access target address of each Nginx access record with each node address of the rest Nginx access records in the Nginx access record set, and determining all nodes of each complete multilevel call chain and corresponding access relations according to the obtained matching result.

According to an enabling aspect of the first aspect of the invention, the method further comprises:

setting corresponding chain identification information according to an access target address of each complete multilevel calling chain, wherein the chain identification information comprises a service name, a project name and/or an access target unique identification of the corresponding multilevel calling chain;

classifying calling relation topology data, corresponding operation quality data, association relation and access quality data among all nodes in the corresponding multilevel calling chain into Nginx instance data, reverse proxy back-end system instance data and project data;

setting a corresponding Nginx instance table, a back-end system instance table and a project function table in a database according to the chain identification information, storing the Nginx instance data in the corresponding Nginx instance table, storing the reverse proxy back-end system instance data in the corresponding back-end system instance table, storing the project data in the corresponding project function table, and adding corresponding labels to each of the Nginx instance table, the back-end system instance table and the project function table.

and synchronizing each Nginx instance table, the backend system instance table and the item function table into an ES index according to the tag.

According to an implementation manner of the first aspect of the present invention, the preset access quality index includes a total access volume, an error volume of the status code exceeding a corresponding preset threshold range, a distribution of the error volume on each URL, a number of logs of which request time exceeds a corresponding item time threshold, an input traffic volume, an output traffic volume, and/or an access volume suspected of having a security risk.

The second aspect of the present invention provides a topology-based Nginx log analysis system, comprising:

the system comprises a requirement determining module, a service analyzing module and a service analyzing module, wherein the requirement determining module is used for determining Nginx log analyzing requirements, and the Nginx log analyzing requirements comprise a unique identifier of a target service and information of an analyzing time period;

the data acquisition module is used for acquiring an Nginx access record set corresponding to the analysis time period from the Kafka cluster according to the unique identifier of the target service; a plurality of Nginx access records are stored in the Kafka cluster, each Nginx access record is obtained by acquiring and analyzing a Nginx access log by a server where a corresponding Nginx instance is located based on a preset Logstash configuration file or a pre-developed acquisition tool, the Nginx access record comprises calling relation topological data and corresponding operation quality data from an access initiator to an access target, the access initiator is a user end, the server where the Nginx instance is located or a reverse proxy back-end system instance, and the access target is the server where the Nginx instance is located or the reverse proxy back-end system instance;

the data analysis module is used for analyzing the calling relation topological data in the Nginx access record set to obtain an association relation among the user sides, the servers where the Nginx instances are located and the reverse proxy backend system instances, and calculating a preset access quality index among association nodes according to the running quality data in the Nginx access record set to obtain corresponding access quality data;

and the topological graph building module is used for building a corresponding topological graph according to the incidence relation and the access quality data, wherein the flow direction of the request data is taken as the direction of each edge of the topological graph, and the attribute of the corresponding edge is set according to the access quality data among the incidence nodes.

According to an implementable manner of the second aspect of the present invention, the data acquisition module comprises:

reading the Nginx access record set by using Logstash.

According to an implementable manner of the second aspect of the invention, the data analysis module comprises:

a determining unit, configured to determine node addresses of all the nginn access records in the nginn access record set from an access initiator to an access target;

and the matching unit is used for matching the access target address of each Nginx access record with each node address of the rest Nginx access records in the Nginx access record set, and determining all nodes of each complete multilevel call chain and corresponding access relations according to the obtained matching result.

According to an enabling manner of the second aspect of the invention, the system further comprises:

the setting module is used for setting corresponding chain identification information according to the access target address of each complete multilevel calling chain, wherein the chain identification information comprises a service name, a project name and/or an access target unique identification of the corresponding multilevel calling chain;

the data classification module is used for classifying the calling relation topology data, the corresponding operation quality data, the association relation and the access quality data among the nodes in the corresponding multilevel calling chain into Nginx example data, reverse proxy back-end system example data and project data;

a data storage module, configured to set a corresponding Nginx instance table, a backend system instance table, and a project function table in a database according to the chain identification information, store the Nginx instance data in the corresponding Nginx instance table, store the reverse proxy backend system instance data in the corresponding backend system instance table, store the project data in the corresponding project function table, and add a corresponding tag to each of the Nginx instance table, the backend system instance table, and the project function table.

and the data synchronization module is used for synchronizing each Nginx instance table, the back-end system instance table and the item function table into an ES index according to the label.

According to an implementation manner of the second aspect of the present invention, the preset access quality index includes a total access amount, an error amount of the status code exceeding a corresponding preset threshold range, a distribution of the error amount on each URL, a log number of the request time exceeding a corresponding item time threshold, an input flow, an output flow, and/or an access amount suspected of having a security risk.

A third aspect of the present invention provides a topology-based Nginx log analysis system, comprising:

the Nginx access log analysis method comprises the steps that the Nginx access log is deployed at an acquisition end of a server where each Nginx instance is located, the acquisition end acquires a Nginx access log based on a preset Logstash configuration file or a pre-developed acquisition tool and analyzes the Nginx access log to obtain a corresponding Nginx access record, and the obtained Nginx access record is sent to a Kafka cluster to be stored; the Nginx access record comprises calling relation topological data and corresponding running quality data from an access initiator to an access target, wherein the access initiator is a user side, a server where an Nginx instance is located or a reverse proxy back-end system instance, and the access target is the server where the Nginx instance is located or the reverse proxy back-end system instance;

the server comprises the Kafka cluster and an analysis device;

the analysis device is used for determining Nginx log analysis requirements, and the Nginx log analysis requirements comprise the unique identification of the target service and information of an analysis time period; acquiring an Nginx access record set corresponding to the analysis time period from the Kafka cluster according to the unique identifier of the target service; analyzing the calling relation topology data in the Nginx access record set to obtain an association relation among the user sides, the servers where the Nginx instances are located and the reverse proxy back-end system instances, and calculating a preset access quality index among association nodes according to the running quality data in the Nginx access record set to obtain corresponding access quality data; and constructing a corresponding topological graph according to the association relation and the access quality data, wherein the flow direction of the request data is taken as the direction of each edge of the topological graph, and the attribute of the corresponding edge is set according to the access quality data among the associated nodes.

According to an implementable aspect of the third aspect of the invention, the analysis device is specifically configured to:

and reading the Nginx access record set by using Logstash.

According to an implementable manner of the third aspect of the present invention, the analysis apparatus is specifically configured to:

determining each node address from an access initiator to an access target of all Nginx access records in the Nginx access record set;

and matching the access target address of each Nginx access record with the node addresses of the rest Nginx access records in the Nginx access record set, and determining all nodes of each complete multilevel call chain and corresponding access relations according to the obtained matching result.

According to an implementable manner of the third aspect of the present invention, the analysis apparatus is further specifically configured to:

setting a corresponding Nginx instance table, a back-end system instance table and a project function table in a database according to the chain identification information, storing the Nginx instance data in the corresponding Nginx instance table, storing the reverse proxy back-end system instance data in the corresponding back-end system instance table, storing the project data in the corresponding project function table, and adding corresponding tags to each of the Nginx instance table, the back-end system instance table and the project function table.

According to an implementation manner of the third aspect of the present invention, the preset access quality index includes a total access volume, an error volume of the status code exceeding a corresponding preset threshold range, a distribution of the error volume on each URL, a number of logs of which request time exceeds a corresponding item time threshold, an input traffic, an output traffic, and/or an access volume suspected of having a security risk.

According to an implementable manner of the third aspect of the present invention, the collecting end includes a logstack configured with the preset logstack configuration file, and the collecting end is specifically configured to execute, by the logstack:

analyzing the acquired Nginx access log to obtain a target field of a Nginx access record, deleting a record of a local request, an access target address and a record of performance data abnormity, and specifying the target field of a type to be converted;

adding the business name of the current log, the identification of the server where the deployed Nginx instance is located and the IP address of the Nginx instance as special fields into the corresponding Nginx access record;

respectively adding the IP address and the access target address of the current Nginx instance into a Nginx service instance set and a back-end system instance set of an external Redis cache;

and converting the user source IP to obtain Chinese names, longitude and latitude of cities, provinces and countries to which the user source IP belongs, and adding the obtained result as a new field to the corresponding Nginx access record.

According to an implementable manner of the third aspect of the present invention, the collecting end is further specifically configured to:

adding codes in a Logstash configuration file to check whether a Nginx access log has malicious access, whether SQL injection attack occurs, whether the length and the content of a parameter are scanned in an attacked tool or a security tool and/or requested to have problems, and adding corresponding fields in corresponding Nginx access records according to the obtained check result.

A fourth aspect of the present invention provides a topology-based Nginx log analysis apparatus, including:

a memory to store instructions; wherein the instructions are for implementing a topology-based Nginx log analysis method as described in any of the above implementable manners;

a processor to execute the instructions in the memory.

A fifth aspect of the present invention is a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a topology-based Nginx log analysis method as set forth in any one of the above-implementable manners.

According to the technical scheme, the invention has the following advantages:

the method comprises the steps that a server where a Nginx instance is located collects Nginx access logs based on a preset Logstash configuration file or a pre-developed collection tool, analyzes call relations and operation quality data in the Nginx access logs, obtains Nginx access records and sends the Nginx access records to a Kafka cluster for storage; extracting a corresponding Nginx access record set from the Kafka cluster according to the Nginx log analysis requirement for analysis so as to obtain the association relation among each user side, the server where each Nginx instance is located and each reverse proxy back-end system instance, calculating a preset access quality index among all association nodes according to corresponding operation quality data so as to obtain access quality data, further constructing a corresponding topological graph according to an analysis result, wherein the flow direction of request data is used as the direction of each edge of the topological graph, and setting the attribute of the corresponding edge according to the access quality data among the association nodes; the invention provides a scheme for analyzing the access relation, the project function composition and the corresponding operation quality between a user and a related system from the overall service display in the form of a topological graph according to a series of Nginx massive access logs distributed in multiple servers, solves the technical problem of how to construct a topological graph reflecting the calling relation and the operation quality change condition between the user and the related system of the overall service based on the Nginx logs, has high-efficiency topological processing performance, and can also form a complete topology formed by service system deployment together with the calling topology between back-end systems; the change of Nginx log analysis requirements can be quickly responded by adjusting the Logstash configuration file or the related information of a pre-developed acquisition tool, and the service global access topological data can be automatically generated and stored from the mass Nginx access logs without recompiling and releasing, so that the method has the advantages of small development workload and convenience in deployment and adjustment; the invention sets the attributes of the corresponding edges according to the access quality data among the associated nodes, thereby directly prompting the found problems on the topological graph, helping technicians comprehensively master the operation quality change condition of the service overall situation through the topological graph, and providing quantitative data support for timely finding, deep analysis, afterwards playback, verification and the like of the network security problems of the distributed system, thereby greatly facilitating the service guarantee work of the distributed system.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without inventive exercise.

Fig. 1 is a flowchart of a method for analyzing a nsinx log based on a topology according to an alternative embodiment of the present invention;

FIG. 2 is a diagram illustrating a specific storage scheme for storing relevant data in Nginx log analysis by using tables according to an alternative embodiment of the present invention;

fig. 3 is a schematic diagram of an example of an application access topology obtained based on the method in fig. 1 according to an alternative embodiment of the present invention;

fig. 4 is a connection block diagram of a topology-based Nginx log analysis system according to an alternative embodiment of the present invention;

fig. 5 is a block diagram illustrating a structural connection of a topology-based Nginx log analysis system according to another alternative embodiment of the present invention;

fig. 6 is a schematic structural diagram of a Nginx log analysis system when a collection end collects a Nginx access log based on a preset Logstash configuration file and analyzes the collected nginnx access log to obtain a corresponding Nginx access record according to an optional embodiment of the present invention.

Reference numerals are as follows:

in FIG. 3, 1-demand determination module; 2-a data acquisition module; 3-a data analysis module; 4-a topological graph building module;

in FIG. 4, 10-collection end; 20-a server side; 201-Kafka cluster; 202-analysis means.

Detailed Description

The embodiment of the invention provides a Nginx log analysis method, a system and equipment based on topology, which are used for solving the technical problem of how to construct a topological graph reflecting the change conditions of the calling relationship and the operation quality between users and relevant systems of the service overall situation based on the Nginx log.

In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the embodiments described below are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

The invention provides a Nginx log analysis method based on topology.

Referring to fig. 1, fig. 1 is a flowchart illustrating a method for analyzing a nsinx log based on a topology according to an embodiment of the present invention.

The Nginx log analysis method based on the topology provided by the embodiment of the invention comprises the steps S1-S4.

Step S1, determining Nginx log analysis requirements, wherein the Nginx log analysis requirements comprise the unique identification of the target service and the information of the analysis time period.

The Nginx log analysis requirement can be determined by receiving analysis requirements initiated by other preset terminals or generating the analysis requirements periodically. The length of the analysis time period is set according to actual requirements.

And S2, acquiring an Nginx access record set corresponding to the analysis time period from the Kafka cluster according to the unique identifier of the target service.

The Kafka cluster stores a plurality of Nginx access records, each Nginx access record is obtained by acquiring and analyzing an Nginx access log by a server where a corresponding Nginx instance is located based on a preset Logstash configuration file or a pre-developed acquisition tool, the Nginx access records comprise call relation topological data and corresponding operation quality data from an access initiator to an access target, the access initiator is a user side, the server where the Nginx instance is located or a reverse proxy back-end system instance, and the access target is the server where the Nginx instance is located or the reverse proxy back-end system instance.

It should be noted that, in order to implement the method in the embodiment of the present invention, the server where the Nginx instance is located may deploy the acquisition end described in the following embodiments of the present invention, so as to implement extraction of the Nginx access record. Details of the extraction are not repeated in this embodiment.

In an implementation manner, the obtaining, according to the unique identifier of the target service, a set of Nginx access records corresponding to the analysis time period from a Kafka cluster includes:

reading the Nginx access record set by using Logstash.

Each of the nginnx access records in the nginnx access record set may be parsed into the required fields for each by Logstash.

And S3, analyzing the calling relation topology data in the Nginx access record set to obtain the association relation among the user sides, the servers where the Nginx instances are located and the reverse proxy backend system instances, and calculating the preset access quality index among the association nodes according to the running quality data in the Nginx access record set to obtain corresponding access quality data.

One Nginx access record corresponds to an actual access request from an access initiator to an access target via Nginx. In a multi-level access link, the access target may also be the access initiator of other Nginx logs. The call relationship topology data of each Nginx access record in the current analysis time period can be analyzed and processed by using a logstack-filter-ruby plug-in, so that the association relationship among each user side, the server where each Nginx instance is located and each reverse proxy back-end system instance is obtained.

Specifically, the judgment can be performed by using the Nginx instance IP, so as to find out all nodes and access relations of the currently complete multi-level call chain from all access records of the current time period. For example, when the access initiator address is the IP of a Nginx instance, a connection is made from that IP to the current Nginx; if the access target address is also a Nginx instance, the current Nginx requests another Nginx instance, namely, a plurality of Nginx layers are provided, the addresses of front-end service instances and back-end service instances of each Nginx are assembled, whether the back-end service is also a Nginx layer or not and the current Nginx layer are synchronously set, and the Nginx layer facing the user side is the first layer; if the access initiator address and the access target address are both rear-end system instance addresses, the rear-end system of the access initiator address calls a system corresponding to the access target address through Nginx; if the access target address is the access initiator address of another Nginx log, the system corresponding to the access target address actually continues to call the Nginx instance to which the corresponding log belongs.

In one implementation manner, the analyzing each invocation relationship topology data in the nginnx access record set includes:

In the embodiment of the invention, the incidence relation among the nodes is determined in an address matching mode, and the method is simple and convenient.

In an implementation manner, the preset access quality index includes a total access amount, an error amount of a status code exceeding a corresponding preset threshold range, a distribution of the error amount on each URL, a log amount of a request time exceeding a corresponding item time threshold, an input flow, an output flow, and/or an access amount suspected of having a security risk.

As a specific implementation manner, the preset access quality index may be calculated by a logstack-filter-ruby plug-in.

As a specific embodiment, it may be checked whether there is a request that satisfies the condition of not less than 3 suspected risks, and the request may be taken as an access suspected of being a security risk. The conditions suspected of being at risk may include:

the user always accesses a specific URL, the range between the IP addresses of the user source IP is unchanged or changed is within a certain threshold, the number of times of operation per second/minute of the user is not within the corresponding threshold range, the average value and the standard deviation of the time difference of adjacent operations of the user are not within the corresponding threshold range, and the request parameters and the rules thereof conform to the preset characteristics.

And S4, constructing a corresponding topological graph according to the association relation and the access quality data, wherein the flow direction of the request data is taken as the direction of each edge of the topological graph, and the attribute of the corresponding edge is set according to the access quality data among the associated nodes.

Wherein, a UI can be developed based on Grafana or some JavaScript interactive graph tools to show the topological graph through the UI.

Furthermore, the access quality data corresponding to the preset access quality index exceeding the alarm threshold value can be marked on the topological graph.

The marking mode can be that different colors, animations, voice prompts and/or connection line thickness effects are set according to different alarm levels so as to emphatically display the position of corresponding data on the topological graph.

The embodiment of the invention can enable related personnel to quickly and intuitively find the service with the running quality problem and the alarm and the influence range thereof by checking the topological graph, is beneficial to improving the efficiency of problem troubleshooting, and fundamentally solves the problem so as to take measures to eliminate the problem in time before the problem is developed to be serious.

Because the data on the topological graph is only a static analysis result of a period of time, the judgment about the global operation health condition of the service can be output by analyzing all alarms in the current selected period of time. For example, if the slow performance of the items without direct calling relationship in one computer room reaches a certain ratio and the error amount exceeds a certain threshold, the machine room infrastructure can be prompted to be checked, and the processing priority of the related items can be prompted according to the alarm amount.

On the UI, the system can also respond to selection, hovering and right-click operation of a mouse on a topological element to support viewing more data, switching to the topology of other services and the like, for example, each node on the topological graph is further provided with more change trend analysis graphs such as the following graph to assist a user to observe the change situation of the operation quality from more dimensions:

(1) Inquiring logs of currently selected nodes in a selected time period from the ES, taking each log as a point, and providing information of time consumption and error of a corresponding request displayed by a scatter diagram;

(2) Providing a histogram to show the number of requests in different performance intervals;

(3) Providing a line graph showing the trend of the chronic energy access amount (i.e., the number of logs whose request time exceeds a threshold specified by the project) over time;

(4) Providing a table to show the access IP with the security risk, the related URL, the type, the occurrence time, the access parameter and the frequency of the security problem which are discovered through analysis; an item has data such as which URLs, relevant description, time of online usage, alarm threshold, and alarms actually occurred.

If the actual topological graph contains more nodes and layers, the user can be allowed to select the maximum allowed node layer times of the business nodes in and out by taking the business as the center on a UI displayed by the topological graph, so that the scale of the specifically displayed topological graph is controlled. The scheme can greatly reduce the development workload, supports the instant adjustment of the configuration of the Logstash and is beneficial to the quick response of the Nginx log analysis requirement.

In one implementation, the method further comprises:

classifying calling relationship topology data, corresponding operation quality data, association relationship and access quality data among all nodes in the corresponding multilevel calling chain into Nginx instance data, reverse proxy back-end system instance data and project data;

setting a corresponding Nginx instance table, a back-end system instance table and a project function table in a database according to the chain identification information, storing the Nginx instance data in the corresponding Nginx instance table, storing the reverse proxy back-end system instance data in the corresponding back-end system instance table, storing the project data in the corresponding project function table, and adding corresponding labels (tag) to each of the Nginx instance table, the back-end system instance table and the project function table.

In the embodiment of the invention, the Nginx example table can support the storage of topology information when one service has multistage Nginx as reverse proxy service. The nginnx instance data stored in the nginnx instance table may include a service to which the nginnx instance belongs, an IP, and an identifier of a server where the nginnx instance belongs. If the service to which the Nginx instance belongs adopts a multi-level Nginx proxy, the Nginx instance data can also include the layer to which the current Nginx belongs, whether the first layer directly receives user access, a front-end service instance set, a back-end service instance set and whether the back-end service of the current Nginx instance is the Nginx or other information. The front-end service instance set is a service instance set for accessing the current Nginx instance, and the back-end service instance set is a set of service instances to be accessed by the current Nginx instance.

In the embodiment of the invention, the back-end system instance table can support the storage of the back-end system instance data of the reverse proxy. The reverse proxy back-end system instance data may include topology information of service instances of the nginnx service reverse proxy, such as the back-end Web, the gRPC, and the like, where the topology information includes a service name, project deployment information, an external communication protocol, an instance address of a called project, a project instance address of a calling current project, corresponding operation quality data, and access quality data. The back-end system instance table may further store a threshold corresponding to the preset access quality index.

In the embodiment of the invention, the project function table can support the storage of project data. The project data may include a name and an access address of a project function (e.g., a URL in an Nginx access log), a service to which the project function belongs, project information called by the project function, corresponding operation quality data, and access quality data. The project function table may also store corresponding access quality indicator thresholds.

For the above embodiment of the present invention, when storing the obtained analysis data, the current topology record may be stored in the database by combining the external service interface with the Nginx instance table, the backend system instance table, and the project function table, and the current system time is set as the operation time. When data is stored, when the number of instances of a back-end system of a service deployment increase or decrease Nginx or an agent thereof and the change of a deployment address are checked, historical records and change time of corresponding topology can be automatically generated in a database, so that a user can be allowed to check the specific change condition of the service topology. When the URLs of the back-end system are stored, the URLs are added only when the URLs do not exist in the table, and the adding time is recorded, so that the functions of a project and the online use time of the functions can be recorded.

Besides the Nginx instance table, the back-end system instance table and the project function table, a service domain name table, a user area access history table and a node operation quality history table can be set to store corresponding data.

For example, the service domain name table may store information of all services related to the topology and domain names thereof, including that the services use a plurality of layers of Nginx proxies;

the user area access history table can be used for storing the access amount of a certain city to a service in a certain time, and the access amount of errors, slow performance, flow and suspected safety problems in the access amount;

the operation quality history table may be used to store operation quality indicator values for each edge of the generated topology graph, including access addresses of a start node (a Nginx instance or a backend system instance) and its accessed target instance (an end node, also a Nginx instance or a backend system instance), a start time of each analysis time interval, main operation quality indicators of the access path, such as an access amount, 4XX and 5XX error amounts, distribution of errors over functions, number of logs (chronic energy) whose request times exceed a threshold specified by an item, input and output traffic, and number of suspected security requests.

It should be noted that the specific contents stored in each table can be appropriately adjusted according to actual needs. Illustratively, fig. 2 is a schematic diagram illustrating a specific storage scheme for storing relevant data in the Nginx log analysis by using tables according to an alternative embodiment of the present invention. The present invention can set the contents that each table should store with reference to the storage scheme shown in fig. 2.

As another possible implementation manner, a graph database may be used for storing the obtained analysis data, the structure of the corresponding topological graph is conceptually and logically consistent with the above storage manner through the table, and the main nodes include: the user source city, the Nginx instance and the back-end system instance, the back-end system instance can be extended into different nodes according to functions, each back-end instance can call other systems through the Nginx, and the attribute of each node can refer to the fields of the table to select data needed in service.

In one implementation, the method further comprises:

Nginx and the ingress of K8S based on Nginx are widely used for proxy backend services, and the content and format of the access logs output by the Nginx and the ingress are highly consistent. In the above embodiment of the present invention, a topological graph is constructed by analyzing a user area, the nginn and its back-end system, and the related system topology of the calling party and the called party of the system, and then matching the indexes of the access amount, error rate, performance, etc. of each part of the topology, so that technicians can conveniently and quickly overview the operation quality of the service overall, quickly find problematic parts and obtain related data, and can also form a complete topology formed by service system deployment together with the calling topology between back-end systems.

Fig. 3 is a schematic diagram illustrating an example of an application access topology obtained based on the method according to the foregoing embodiment of the present invention according to an alternative embodiment of the present invention.

As shown in fig. 3, when the access quality data between the associated nodes is taken as the attribute of the edge, the access quality data is presented in a format like "total 268/slow 12/error 5/ampere 13". "total 268/slow 12/error 5/security 13" means that there are a total of 268 access logs between two nodes in a selected time period, where 12 requests have performance exceeding a threshold, 5 requests have errors, and 13 requests are suspected of being at security risk. It should be noted that other indexes (for example, traffic) may be added to the required service in the format, and if there is no problem, the corresponding index is set to 0.

The invention also provides a Nginx log analysis system based on the topology.

Referring to fig. 4, fig. 4 is a connection block diagram illustrating a structure of a topology-based Nginx log analysis system according to an embodiment of the present invention.

The Nginx log analysis system based on topology provided by the embodiment of the invention comprises:

the system comprises a requirement determining module 1, a service analyzing module and a service analyzing module, wherein the requirement determining module is used for determining Nginx log analyzing requirements, and the Nginx log analyzing requirements comprise a unique identifier of a target service and information of an analyzing time period;

the data acquisition module 2 is configured to acquire an Nginx access record set corresponding to the analysis time period from the Kafka cluster according to the unique identifier of the target service; a plurality of Nginx access records are stored in the Kafka cluster, each Nginx access record is obtained by acquiring and analyzing an Nginx access log through a server where a corresponding Nginx instance is located based on a preset Logstash configuration file or a pre-developed acquisition tool, the Nginx access record comprises call relation topological data and corresponding operation quality data from an access initiator to an access target, the access initiator is a user end, the server where the Nginx instance is located or a reverse proxy back-end system instance, and the access target is the server where the Nginx instance is located or the reverse proxy back-end system instance;

a data analysis module 3, configured to analyze each calling relationship topology data in the Nginx access record set to obtain an association relationship between each user side, a server where each Nginx instance is located, and each reverse proxy backend system instance, and calculate a preset access quality index between each association node according to each operation quality data in the Nginx access record set to obtain corresponding access quality data;

and the topological graph building module 4 is used for building a corresponding topological graph according to the association relation and the access quality data, wherein the flow direction of the request data is taken as the direction of each edge of the topological graph, and the attribute of the corresponding edge is set according to the access quality data among the associated nodes.

In an implementable manner, the data acquisition module 2 comprises:

reading the Nginx access record set by using Logstash.

In an implementable manner, the data analysis module 3 comprises:

In one enabling form, the system further includes:

the setting module is used for setting corresponding chain identification information according to the access target address of each complete multilevel calling chain, wherein the chain identification information comprises a business name, a project name and/or an access target unique identification of the corresponding multilevel calling chain;

In one enabling form, the system further includes:

The invention also provides a Nginx log analysis system based on the topology.

Referring to fig. 5, fig. 5 is a block diagram illustrating a structural connection of a topology-based Nginx log analysis system according to another alternative embodiment of the present invention.

The embodiment of the invention provides a Nginx log analysis system based on topology, which comprises:

the method comprises the steps that an acquisition end 10 is deployed in a server where each Nginx instance is located, the acquisition end 10 acquires Nginx access logs based on a preset Logstash configuration file or a pre-developed acquisition tool and analyzes the Nginx access logs to obtain corresponding Nginx access records, and the obtained Nginx access records are sent to a Kafka cluster 201 to be stored; the Nginx access record comprises calling relation topological data and corresponding running quality data from an access initiator to an access target, wherein the access initiator is a user side, a server where an Nginx instance is located or a reverse proxy back-end system instance, and the access target is the server where the Nginx instance is located or the reverse proxy back-end system instance;

the server 20 comprises the Kafka cluster 201 and an analysis device 202;

the analysis device 202 is configured to determine a Nginx log analysis requirement, where the Nginx log analysis requirement includes a unique identifier of a target service and information of an analysis time period; acquiring an Nginx access record set corresponding to the analysis time period from the Kafka cluster 201 according to the unique identifier of the target service; analyzing the calling relation topology data in the Nginx access record set to obtain an association relation among the user sides, the servers where the Nginx instances are located and the reverse proxy back-end system instances, and calculating a preset access quality index among association nodes according to the running quality data in the Nginx access record set to obtain corresponding access quality data; and constructing a corresponding topological graph according to the association relation and the access quality data, wherein the flow direction of the request data is taken as the direction of each edge of the topological graph, and the attribute of the corresponding edge is set according to the access quality data among the associated nodes.

In an implementation manner, the analysis apparatus 202 is specifically configured to:

and reading the Nginx access record set by using Logstash.

In an implementation manner, the analysis apparatus 202 is further specifically configured to:

and synchronizing each Nginx instance table, the backend system instance table and the item function table into an ES index according to the tags.

Referring to fig. 6, fig. 6 is a schematic structural diagram of a Nginx log analysis system when the acquisition end 10 acquires a Nginx access log based on a preset Logstash configuration file and analyzes the nginnx access log to obtain a corresponding Nginx access record according to an optional embodiment of the present invention.

As shown in fig. 6, a Logstash and an external Redis cache configured with the preset Logstash configuration file are deployed in each machine room (i.e., a server where the ngix instance is located), and the Logstash and the external Redis cache form an acquisition end 10 of the server where the ngix instance is located. The server 20 is provided with a DB library for storing a Nginx instance table, a backend system instance table, and a project function table, and an ES library for synchronizing each of the Nginx instance table, the backend system instance table, and the project function table into an ES index.

In an implementation manner, the acquisition end 10 is specifically configured to perform, by Logstash:

respectively adding the IP address and the access target address of the current Nginx instance into an Nginx service instance set and a back-end system instance set of an external Redis cache;

In this embodiment, the Logstash configured with the preset Logstash configuration file is deployed in the server where each Nginx instance is located to perform Nginx access log collection and analysis. The functions of the acquisition terminal 10 according to this embodiment can be realized by editing the configuration file of the Logstash until the processed result is output to the Kafka cluster 201. By adjusting the logstack configuration file, the change of the Nginx log analysis requirement can be quickly responded, and the service global access topological data can be automatically generated and stored from the mass Nginx access logs without recompiling and releasing, so that the method has the advantages of small development workload and convenience in deployment and adjustment.

The user source IP is converted to obtain Chinese names, the longitude and the latitude of cities, provinces and countries to which the user source IP belongs, and the data obtained by conversion can reflect the access quality of users in different areas, so that the conditions of whether an operator network is fully accessed, whether the bandwidth is sufficient, whether the CDN needs to be started and the like are further checked, a data basis is provided for improving the access quality, preparation is made for the development of services in advance, and user complaints caused by poor machine room deployment are avoided.

As logstack cannot output the name of the chinese region and the output field is not controllable, logstack-filter-geoip can be developed secondarily for the defect as a way to implement, so as to implement the function of logstack of the acquisition end 10 in this embodiment to convert the user source IP.

Specifically, a file with a path name of geo-rb under the Logstash directory may be modified, and the main modified contents may include:

according to the characteristics that users applying the Internet often have region aggregation and repeated access, the corresponding relation between the IP of the LRU cache user and the region is constructed and started, and the time consumption for searching an IP library is reduced by multiplexing;

in combination with the requirement of data analysis, specifying a required regional field name, wherein the regional field name can comprise country, province (regional name), city and/or longitude and latitude;

and modifying the filter method, utilizing GeoLite2-City.mmdb to support an analysis library of an IPv6 address above a corresponding code block so as to obtain a corresponding city object according to the analysis of the user IP, and then obtaining required fields such as Chinese values, longitude and latitude, postal codes and the like of countries and regions (provinces) from the object.

In an implementation manner, the collecting end 10 is further specifically configured to:

As a specific implementation manner, to implement the operation of the acquisition end 10 in the above embodiment, the Logstash in the acquisition end 10 specifically executes:

according to log _ format configuration of Nginx, analyzing by adopting a csv plug-in of Logstash to obtain values of fields such as a user source, a back-end system instance address of a reverse proxy, a request and corresponding performance, input/output flow, a state code, a URL of the request, a request parameter, a forwarding service address and a request length;

calling drop { } to delete the record of the local request, the record of the back-end system instance address and the performance data exception of the reverse proxy; using a multicast plug-in to specify fields (including performance, flow and date and time) of types needing to be converted, and deleting the fields which are not needed;

adding the service name of the current log, the deployment machine room and the IP of the Nginx instance as special fields into the log record;

respectively adding the IP of the current Nginx instance and the address of the back-end system instance of the reverse proxy into the Nginx service instance set and the back-end system instance set cached by the external Redis by using a logstack-filter-ruby plug-in;

enabling a geo plug-in, designating a local storage path of an IP library GeoLite2-City.mmdb to be used, converting a user source IP through logstack-filter-geo developed for the second time, and adding Chinese names, longitude and latitude of cities, provinces and countries to which the IP belongs as new fields into an output record;

and adding a code in the Logstash configuration file by using a Logstash-filter-ruby plug-in to check whether the request URL and the request parameter of the log have malicious access.

Wherein, the content and the code of the examination can be adjusted at any time according to the actual need. If any problem is found, two fields are added in the current log record to respectively indicate the type and the description of the found problem, and a tag is added to distinguish the current log record from other logs without security risks.

Adding codes in the Logstash configuration file to check whether the nginnx access log has malicious access, whether SQL injection attack occurs, whether the length and content of parameters scanned and requested by an attack tool or a security tool have problems, and specific checking details may include:

(1) Checking whether SQL injection attack occurs: checking whether partial contents of SQL sentences, SQL functions, built-in databases of database services and names or version numbers of tables appear in logs with response status codes of 200 or 5XX, and if yes, adding a field description to the current record to indicate SQL injection attack and corresponding back-end system response information;

(2) Checking whether there is malicious access behavior: checking whether the request URL and the parameters thereof contain at least one file in a file, HTTP relative or absolute path mode, if so, further checking whether the directory name, the file name and the file extension contained in the request URL and the parameters thereof are in a batch of sensitive paths and file lists, and recording the checking result, the request path and the parameters; checking whether the request URL and the parameters thereof transmit the content assisting certain code execution, and judging whether the corresponding access is reflected, loaded with any type and code or tries to load the code which can be executed by a certain shell and execute by using the running platform according to the checking result;

(3) Check if it is being scanned by an attack tool or security tool: determining whether the business system is being scanned by a tool of an abnormal user by checking whether a part of the content of the current log agent field contains one of a group of suspected problem tools;

(4) Check length and content of request parameters: checking whether the length of the request parameter exceeds a corresponding preset threshold value, and if so, indicating that the corresponding request is possibly illegal or belongs to an abnormal request; if the content contains hexadecimal characters, the configuration problem of SSL, character coding and decoding and the like possibly exists in Nginx, and the specific problem can be correspondingly judged when some hexadecimal characters are detected;

(5) The result data of each project is written into the respective theme of the Kafka cluster 201 through the output plug-in of Logstash, then according to whether the log records the tag with the security risk, the user IP, the requested URL, the parameters, the occurrence time and the response status code of each log with suspected security behavior are extracted, and the extracted data is written into a database for storing the information, so that the alarm is convenient to give an alarm in time.

After the above operation of the acquisition end 10, each nginnx access record already contains the call relationship from the operation initiator (nginnx, the backend system instance or the area where the user is located) to the access target and the operation quality information thereof. After the access logs of all the Nginx instances are transferred to the Kafka cluster 201, the data actually include the Nginx where each user area accesses all the services, topology data called among relevant backend systems, and values of indexes such as access amount, performance, success rate and the like of each edge in the topology within a certain time. The server 20 further uses the logstack to extract and store topology composition and related operation quality information according to the need of topology update, so as to be used when displaying the topology, alarming the problem needing attention and playing back the topology in real time.

In another implementation manner, the acquisition end 10 is specifically configured to acquire and analyze an Nginx access log through a pre-developed acquisition tool. For example, the collection tool may be developed by using a language such as Go, and then deployed to a server where each Nginx instance is located, and then the collection tool periodically calculates topology composition and operation quality data according to the access log of each Nginx instance, and then sends the topology composition and operation quality data to the server 20 for processing.

The advantages of acquiring and analyzing the Nginx access log by the pre-developed acquisition tool include: the constraint of the Logstash can be broken through to realize more requirements; the tool may open the HTTP service to support performing required control operations on the Nginx according to the real-time instruction of the server 20 at runtime, for example, limiting excessive access according to user configuration, checking log contents corresponding to more security risk types and more configuration errors, modifying the Nginx configuration, reloading and validating, and the like; logstash is not needed any more, and deployment can be simplified. The scheme changes mainly in the Nginx log processing part, including:

reading a configuration file of Nginx on a current server, obtaining log _ format of a Nginx access log and a back-end instance address (possibly still being the Nginx instance address) of a reverse proxy, obtaining a service domain name and a machine room to which the service domain name belongs according to the configuration file of a tool, reading an IP (Internet protocol) of the Nginx instance, and submitting the information to a server 20, wherein the server 20 can store history information such as change content and time of corresponding services for subsequent verification when detecting the deployment change of each service;

reading Nginx access log file records in batches, analyzing required fields according to log _ format, and deleting unneeded log records according to the same logic;

and obtaining the city, province, country and longitude and latitude of the user source according to the address of the access initiator by utilizing an IP library and an analysis tool provided by GeoIP2 and matching with a locally constructed LRU cache, and adding a result field into a corresponding log record.

The invention also provides a Nginx log analysis device based on topology, which comprises:

a memory to store instructions; wherein the instructions are for implementing a topology based Nginx log analysis method as described in any of the above embodiments;

a processor to execute the instructions in the memory.

The present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the topology-based Nginx log analysis method as described in any one of the above embodiments.

It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the system, the device and the module described above may refer to the corresponding processes in the foregoing method embodiments, and the specific beneficial effects of the system, the device and the module described above may refer to the corresponding beneficial effects in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus, and method may be implemented in other ways. For example, the above-described system embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not executed.

The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

In addition, functional modules in the embodiments of the present invention may be integrated into one processing module, or each module may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode.

The integrated module, if implemented in the form of a software functional module and sold or used as a separate product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A topology-based Nginx log analysis method is characterized by comprising the following steps:

acquiring an Nginx access record set corresponding to the analysis time period from the Kafka cluster according to the unique identifier of the target service; a plurality of Nginx access records are stored in the Kafka cluster, each Nginx access record is obtained by acquiring and analyzing a Nginx access log by a server where a corresponding Nginx instance is located based on a preset Logstash configuration file or a pre-developed acquisition tool, the Nginx access record comprises calling relation topological data and corresponding operation quality data from an access initiator to an access target, the access initiator is a user end, the server where the Nginx instance is located or a reverse proxy back-end system instance, and the access target is the server where the Nginx instance is located or the reverse proxy back-end system instance;

2. The topology based Nginx log analysis method according to claim 1, wherein the obtaining a Nginx access record set corresponding to the analysis time period from a Kafka cluster according to the unique identifier of the target service comprises:

reading the Nginx access record set by using Logstash.

3. The topology based Nginx log analysis method according to claim 1, wherein the analyzing each of the call relation topology data in the Nginx access record set comprises:

4. The topology based Nginx log analysis method of claim 3, further comprising:

5. The topology based Nginx log analysis method of claim 4, further comprising:

6. The topology-based Nginx log analysis method of claim 1, wherein the preset access quality indicators include total access volume, error volume for which status codes exceed corresponding preset threshold ranges, distribution of the error volume on each URL, number of logs for which request time exceeds corresponding project time threshold, input traffic, output traffic, and/or access volume suspected of having a security risk.

7. A topology based Nginx log analysis system, comprising:

8. A topology based Nginx log analysis system, comprising:

the server comprises the Kafka cluster and an analysis device;

9. The topology based Nginx log analysis system according to claim 8, wherein the collecting end includes a Logstash configured with the preset Logstash configuration file, and the collecting end is specifically configured to perform, by the Logstash:

analyzing the acquired Nginx access log to obtain a target field of a Nginx access record, deleting a record of a local request, an access target address and a record of performance data abnormity, and specifying a target field of a type to be converted;

10. The topology-based Nginx log analysis system according to claim 9, wherein the collection end is further specifically configured to:

adding codes in a Logstash configuration file to check whether a Nginx access log has malicious access, whether SQL injection attack occurs, whether the length and the content of a parameter are scanned and/or requested by an attacked tool or a security tool have problems, and adding corresponding fields in corresponding Nginx access records according to an obtained check result.

11. A topology-based Nginx log analysis device, comprising:

a memory to store instructions; wherein the instructions are for implementing a topology based Nginx log analysis method as claimed in any of claims 1-6;

a processor to execute the instructions in the memory.

12. A computer-readable storage medium, characterized in that a computer program is stored thereon, which computer program, when being executed by a processor, implements the topology based Nginx log analysis method according to any one of claims 1 to 6.