CN112256648A - Method and device for collecting and tracking log behaviors based on Nginx - Google Patents
Method and device for collecting and tracking log behaviors based on Nginx Download PDFInfo
- Publication number
- CN112256648A CN112256648A CN202011151710.6A CN202011151710A CN112256648A CN 112256648 A CN112256648 A CN 112256648A CN 202011151710 A CN202011151710 A CN 202011151710A CN 112256648 A CN112256648 A CN 112256648A
- Authority
- CN
- China
- Prior art keywords
- log
- generating
- nginx
- unique code
- user unique
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000006399 behavior Effects 0.000 title claims abstract description 26
- 238000000034 method Methods 0.000 title claims abstract description 19
- 238000004140 cleaning Methods 0.000 claims abstract description 11
- 230000002688 persistence Effects 0.000 claims abstract description 7
- 230000005540 biological transmission Effects 0.000 claims description 9
- 238000012216 screening Methods 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 4
- 230000008676 import Effects 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 230000010354 integration Effects 0.000 abstract description 3
- 239000012634 fragment Substances 0.000 abstract 1
- 230000000694 effects Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/288—Entity relationship models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
Abstract
The invention relates to the technical field of big data, in particular to a method and a device for acquiring and tracking behaviors based on a Nginx log, wherein the method mainly comprises the following steps: generating a user unique code; modifying a configuration file of Nginx; periodically scanning and collecting the fragmented log records of Nginx; generating an entity class record list, and then cleaning the fragment log record to record required information in the entity class record list; obtaining variable names in the entity class through Java reflection, generating a Mysql database structure field, and then importing the data in the entity class record list into a Mysql database for persistence; and generating a configuration table and a core report, and displaying the core report. The invention has the advantages of simple integration, high customization and low invasiveness.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a method and a device for acquiring and tracking behaviors based on a Nginx log.
Background
Behavior collection and tracking refers to collecting user behaviors according to user operations, performing a series of tracking of each behavior of a user, and collecting and analyzing a large amount of data. The method can well perform user analysis according to user behaviors and tracking, and perform product optimization or integrate the product into a recommendation system according to an analysis result. Such as the current statistical platforms of union, growth, etc. However, most of the platforms are collected aiming at the user terminal and are bound with the UI control, development of the mobile terminal and the front terminal is complex, development cost is high, the platforms are third-party platforms, customization is low, and the platforms cannot be integrated into own systems for customization development.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a method and a device for acquiring and tracking behaviors based on a Nginx log, and the method and the device have the advantages of simple integration, high customization and low invasiveness.
The technical purpose of the invention is realized by the following technical scheme, and the method for acquiring and tracking the behavior based on the Nginx log comprises the following steps:
acquiring a necessary transmission parameter, and generating a user unique code in a DES encryption and decryption mode;
modifying a configuration file of the Nginx based on the essential transmission parameters and the user unique code, and printing request content and outputting the request content to a log file;
setting scanning time to periodically scan and collect Nginx slicing log records;
defining a log parameter template character string, generating an entity class record list, and then cleaning the fragmented log records to record required information in the entity class record list;
obtaining variable names in the entity class through Java reflection, generating a Mysql database structure field, and then importing the data in the entity class record list into a Mysql database for persistence;
generating a configuration table based on the Mysql database, and setting a corresponding relation in the configuration table to generate a core report;
and acquiring the core report, and screening and displaying the core report.
In one embodiment, the step of generating the user unique code further specifically includes the following steps:
the client side obtains the request parameters and sends the request parameters to the server side;
the server side judges whether the client side has the user unique code;
and if not, generating the user unique code in a DES encryption and decryption mode according to the user login state, userld, the timestamp, the random code and the client identifier, and returning the user unique code to the client.
In one embodiment, the step of setting the scanning time further comprises the following steps:
and setting the scanning time in a mode of a java asynchronous thread timing task.
In one embodiment, the request content comprises a request parameter, a timestamp, a user unique code, a parameter value in a Get request, an interface link, an HTTP response code and a user terminal browser.
In one embodiment, the step of generating the core report further includes the steps of:
generating the core report of the data to be put in storage based on the Mysql database;
and generating the core report of the real-time data based on the interface link, the calling time and the calling status code of the user registration interface.
The invention also provides a device for acquiring and tracking the log behavior based on the Nginx, which comprises the following components:
the information acquisition module is used for acquiring the necessary transmission parameters and generating a user unique code in a DES encryption and decryption mode;
the information import module is used for modifying the configuration file of the Nginx according to the must-pass parameter and the user unique code and printing the request content into a log file;
the log acquisition module is used for setting scanning time so as to periodically scan and acquire the fragmented log records of the Nginx;
the log cleaning module is used for defining a log parameter template character string, generating an entity type record list, and cleaning the segmented log records so as to record required information in the entity type record list;
the log importing module is used for acquiring variable names in the entity class through Java reflection, generating a Mysql database structure field and then importing the data of the entity class record list into the Mysql database for persistence;
the log statistical module is used for generating a configuration table according to the Mysql database and setting the corresponding relation in the configuration table to generate a core report;
and the log display module is used for acquiring the core report and screening and displaying the core report.
In one embodiment, the information obtaining module further includes:
the port information extraction submodule is used for the client to obtain a request head parameter and send the request head parameter to the server;
the port information judgment submodule is used for the server side to judge whether the client side has the user unique code;
if the server side judges that the client side does not have the user unique code, the port information judgment submodule generates the user unique code in a DES encryption and decryption mode according to the user login state, userld, the timestamp, the random code and the client side identifier, and returns the user unique code to the client side.
In one embodiment, the log collection module further includes:
and the time setting submodule is used for setting the scanning time in a mode of executing the java asynchronous thread timing task.
In one embodiment, the log statistics module further comprises:
the warehousing data generation submodule is used for generating the core report of the warehousing data according to the Mysql database;
and the real-time data generation submodule is used for generating a core report of the real-time data according to the interface link, the calling time and the calling state code of the user registration interface.
The invention has the following beneficial effects:
firstly, Nginx log information is used for data acquisition, a set of platform integrating cleaning, importing, statistical analysis and displaying is provided, the cost is low, and the effects of simple integration, high customization and low invasiveness are achieved;
secondly, the report forms of various data can be visually displayed in real time, so that the user can conveniently call the data.
Drawings
FIG. 1 is a flow chart of the steps of a method of the present invention for Nginx-based log behavior acquisition and tracking;
FIG. 2 is a block diagram of the structure of the Nginx log behavior-based acquisition and tracking device of the invention.
Detailed Description
The invention is described in detail below with reference to the figures and examples.
The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention, and the components related to the present invention are only shown in the drawings rather than drawn according to the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.
As shown in fig. 1, a method for acquiring and tracking based on a Nginx log behavior includes the following steps:
and S1, acquiring the necessary transmission parameters, and generating the user unique code in a DES encryption and decryption mode.
Specifically, the specific step of generating the user unique code is as follows: the client side obtains the request parameters and sends the request parameters to the server side; the server side judges whether the client side has the unique user code; and if not, generating a user unique code in a DES encryption and decryption mode according to the user login state, userld, the timestamp, the random code and the client identifier, and returning the user unique code to the client.
The encryption and decryption of the DES encryption key are realized by executing the following codes:
s2, based on the must-pass parameter and the user unique code, modifying the configuration file of Nginx, and printing the request content to output to a log file.
Specifically, the request content comprises a request parameter, a timestamp, a user unique code, a parameter value in a Get request, an interface link, an HTTP response code and a user terminal browser.
And S3, setting scanning time to periodically scan and acquire the slicing log records of the Nginx.
Specifically, by setting a java asynchronous thread timing task method, the fragmented log records of the Nginx are periodically scanned according to a set time (for example, scanning once in 10 minutes), the acquired line number is automatically recorded, and the next scanning automatically skips the corresponding line number and acquires the next time.
S4, defining a log parameter template character string, generating an entity class record list, and cleaning the fragmented log records to record required information in the entity class record list.
Specifically, when defining a log parameter template character string, parameters required by a user can be screened and defined into a parameter template character string according to a self-defined Nginx log template parameter, and if a post request parameter, a user unique code, a timestamp, a parameter value in a Get request, an interface link and the like are input, the parameter name can be sent to the value of the corresponding parameter name of the acquired log through java reflection according to the name.
S5, obtaining variable names in the entity class through Java reflection, generating Mysql database structure fields, and then importing the data of the entity class record list into the Mysql database for persistence.
S6, generating a configuration table based on the Mysql database, and setting the corresponding relation in the configuration table to generate a core report.
Specifically, the step of generating the core report further includes the following steps:
generating a core report of the data to be put in a database based on the Mysql database;
and generating a core report of the real-time data based on the interface link, the calling time and the calling status code of the user registration interface.
And (3) core report of the data to be put in storage: the method comprises the steps of generating a configuration table recorded by a database, setting a corresponding relation between an interface link and a name, defining a linear relation between the interface link and the interface link, combining input data, generating a data report of an interface process step, counting behavior data and tracking behavior data through a user unique code, and generating a core report of the database in various modes.
Core report of real-time data: and generating a core report of real-time data by using multiple dimensions selected by the acquired data, wherein if a newly added user is added, the core report of the corresponding data can be realized aiming at the service of each interface through the interface link, the timestamp and the calling status code of the registered interface.
And S7, acquiring the core report, and screening and displaying the core report.
Specifically, the generated data can be dynamically obtained in real time by screening the database field of the collected data as a dimension and combining a pattern displaying mode, and statistics and display are carried out.
As shown in fig. 2, the present invention further provides a device for acquiring and tracking behavior based on a Nginx log, including:
the information acquisition module is used for acquiring the necessary transmission parameters and generating a user unique code in a DES encryption and decryption mode;
the information import module is used for modifying the configuration file of the Nginx according to the requisite transmission parameters and the user unique code and printing the request content into a log file;
the log acquisition module is used for setting scanning time so as to periodically scan and acquire the fragmented log records of the Nginx;
the log cleaning module is used for defining a log parameter template character string, generating an entity type record list, and cleaning the segmented log records so as to record required information in the entity type record list;
the log importing module is used for acquiring variable names in the entity classes through Java reflection, generating Mysql database structure fields and importing the data of the entity class recording list into the Mysql database for persistence;
the log statistical module is used for generating a configuration table according to the Mysql database and setting the corresponding relation in the configuration table to generate a core report;
and the log display module is used for acquiring the core report and screening and displaying the core report.
Specifically, the information acquisition module further includes a port information extraction submodule and a port information judgment submodule.
The port information extraction submodule is used for the client to obtain a request head parameter and send the request head parameter to the server;
the port information judgment submodule is used for the server side to judge whether the client side has the user unique code;
if the server side judges that the client side does not have the user unique code, the port information judgment submodule generates the user unique code in a DES encryption and decryption mode according to the user login state, userld, the timestamp, the random code and the client side identifier, and returns the user unique code to the client side.
The log collection module also comprises a time setting submodule.
The time setting submodule is used for setting scanning time in a mode of executing a java asynchronous thread timing task.
The log statistic module also comprises a warehousing data generation submodule and a real-time data generation submodule.
The warehousing data generation submodule is used for generating a core report of warehousing data according to the Mysql database;
and the real-time data generation submodule is used for generating a core report of the real-time data according to the interface link, the calling time and the calling state code of the user registration interface.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (9)
1. A method for collecting and tracking log behaviors based on Nginx is characterized by comprising the following steps:
acquiring a necessary transmission parameter, and generating a user unique code in a DES encryption and decryption mode;
modifying a configuration file of the Nginx based on the essential transmission parameters and the user unique code, and printing request content and outputting the request content to a log file;
setting scanning time to periodically scan and collect Nginx slicing log records;
defining a log parameter template character string, generating an entity class record list, and then cleaning the fragmented log records to record required information in the entity class record list;
obtaining variable names in the entity class through Java reflection, generating a Mysql database structure field, and then importing the data in the entity class record list into a Mysql database for persistence;
generating a configuration table based on the Mysql database, and setting a corresponding relation in the configuration table to generate a core report;
and acquiring the core report, and screening and displaying the core report.
2. The method for Nginx log behavior-based collection and tracking according to claim 1, wherein the step of generating a user unique code further comprises the steps of:
the client side obtains the request parameters and sends the request parameters to the server side;
the server side judges whether the client side has the user unique code;
and if not, generating the user unique code in a DES encryption and decryption mode according to the user login state, userld, the timestamp, the random code and the client identifier, and returning the user unique code to the client.
3. The method for Nginx log behavior-based acquisition and tracking according to claim 1, wherein the step of setting the scan time further comprises the steps of:
and setting the scanning time in a mode of a java asynchronous thread timing task.
4. The method for Nginx log behavior-based collection and tracking of claim 1, wherein: the request content comprises request parameters, a timestamp, a user unique code, parameter values in a Get request, an interface link, an HTTP response code and a user terminal browser.
5. The method for Nginx log behavior-based collection and tracking according to claim 1, wherein the step of generating a core report further comprises the steps of:
generating the core report of the data to be put in storage based on the Mysql database;
and generating the core report of the real-time data based on the interface link, the calling time and the calling status code of the user registration interface.
6. An apparatus for Nginx log behavior-based collection and tracking, comprising:
the information acquisition module is used for acquiring the necessary transmission parameters and generating a user unique code in a DES encryption and decryption mode;
the information import module is used for modifying the configuration file of the Nginx according to the must-pass parameter and the user unique code and printing the request content into a log file;
the log acquisition module is used for setting scanning time so as to periodically scan and acquire the fragmented log records of the Nginx;
the log cleaning module is used for defining a log parameter template character string, generating an entity type record list, and cleaning the segmented log records so as to record required information in the entity type record list;
the log importing module is used for acquiring variable names in the entity class through Java reflection, generating a Mysql database structure field and then importing the data of the entity class record list into the Mysql database for persistence;
the log statistical module is used for generating a configuration table according to the Mysql database and setting the corresponding relation in the configuration table to generate a core report;
and the log display module is used for acquiring the core report and screening and displaying the core report.
7. The apparatus for Nginx log behavior-based collection and tracking according to claim 6, wherein the information obtaining module further comprises:
the port information extraction submodule is used for the client to obtain a request head parameter and send the request head parameter to the server;
the port information judgment submodule is used for the server side to judge whether the client side has the user unique code;
if the server side judges that the client side does not have the user unique code, the port information judgment submodule generates the user unique code in a DES encryption and decryption mode according to the user login state, userld, the timestamp, the random code and the client side identifier, and returns the user unique code to the client side.
8. The apparatus for Nginx log behavior-based collection and tracking according to claim 6, wherein the log collection module further comprises:
and the time setting submodule is used for setting the scanning time in a mode of executing the java asynchronous thread timing task.
9. The apparatus for Nginx log behavior-based collection and tracking according to claim 6, wherein the log statistics module further comprises:
the warehousing data generation submodule is used for generating the core report of the warehousing data according to the Mysql database;
and the real-time data generation submodule is used for generating a core report of the real-time data according to the interface link, the calling time and the calling state code of the user registration interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011151710.6A CN112256648A (en) | 2020-10-26 | 2020-10-26 | Method and device for collecting and tracking log behaviors based on Nginx |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011151710.6A CN112256648A (en) | 2020-10-26 | 2020-10-26 | Method and device for collecting and tracking log behaviors based on Nginx |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112256648A true CN112256648A (en) | 2021-01-22 |
Family
ID=74262913
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011151710.6A Pending CN112256648A (en) | 2020-10-26 | 2020-10-26 | Method and device for collecting and tracking log behaviors based on Nginx |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112256648A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113312376A (en) * | 2021-05-21 | 2021-08-27 | 福建天泉教育科技有限公司 | Method and terminal for real-time processing and analysis of Nginx logs |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090055443A1 (en) * | 2007-08-23 | 2009-02-26 | International Business Machines Corporation | Recording a Log of Operations |
| CN103778244A (en) * | 2014-02-11 | 2014-05-07 | 五八同城信息技术有限公司 | Automatic report analytical method based on user behavior logs |
| CN107977473A (en) * | 2017-12-28 | 2018-05-01 | 政采云有限公司 | The search method and system of distributed system journal based on Logback |
-
2020
- 2020-10-26 CN CN202011151710.6A patent/CN112256648A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090055443A1 (en) * | 2007-08-23 | 2009-02-26 | International Business Machines Corporation | Recording a Log of Operations |
| CN103778244A (en) * | 2014-02-11 | 2014-05-07 | 五八同城信息技术有限公司 | Automatic report analytical method based on user behavior logs |
| CN107977473A (en) * | 2017-12-28 | 2018-05-01 | 政采云有限公司 | The search method and system of distributed system journal based on Logback |
Non-Patent Citations (3)
| Title |
|---|
| DYF0333: "用nginx访问日志记录mysql数据库中的用户id", pages 1 - 4, Retrieved from the Internet <URL:https://github.com/dyf0333/nginx/blob/master/%E7%94%A8nginx%E8%AE%BF%E9%97%AE%E6%97%A5%E5%BF%97%E8%AE%B0%E5%BD%95mysql%E6%95%B0%E6%8D%AE%E5%BA%93%E4%B8%AD%E7%9A%84%E7%94%A8%E6%88%B7id.md> * |
| HAOBOOK: "nginx filter模块实践", pages 1 - 2, Retrieved from the Internet <URL:https://haobook.readthedocs.io/zh-cn/latest/periodical/201608/haoyankai.html> * |
| PLAYINJAVA: "Spark 从零到开发（八）nginx日志清洗并持久化实战", pages 1 - 5, Retrieved from the Internet <URL:https://juejin.cn/post/6844903661500366862> * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113312376A (en) * | 2021-05-21 | 2021-08-27 | 福建天泉教育科技有限公司 | Method and terminal for real-time processing and analysis of Nginx logs |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107592346B (en) | User classification method based on user behavior analysis | |
| CN107688530A (en) | Method for testing software and device | |
| CN112134785A (en) | Information processing method, client and system in network security level protection | |
| CN108491210B (en) | Code continuous integrating method, apparatus, system and server | |
| CN113094262B (en) | Method for testing production data based on database and table division | |
| CN112256648A (en) | Method and device for collecting and tracking log behaviors based on Nginx | |
| CN112905935A (en) | Page recording method, page recording animation generation method, equipment and storage medium | |
| CN111581067A (en) | Data acquisition method and device | |
| CN110209575A (en) | Test document generation method, device, electronic equipment and storage medium | |
| CN113986954B (en) | User event acquisition method and device, intelligent terminal and readable storage medium | |
| CN111046382A (en) | Database auditing method, device, storage medium and device | |
| CN113672497B (en) | Method, device and equipment for generating non-buried point event and storage medium | |
| CN113115107B (en) | Handheld video acquisition terminal system based on 5G network | |
| CN115063103A (en) | Schedule management method, schedule management system, storage medium and electronic equipment | |
| CN110941608B (en) | Method, device and equipment for generating buried point analysis and funnel analysis report | |
| CN112905558A (en) | Report system implementation method and system based on database configuration | |
| CN112364267A (en) | Front-end data acquisition method and device | |
| CN114691837B (en) | Insurance business data processing method and processing system based on big data | |
| CN116016629B (en) | Front-end page event extraction method | |
| CN111581931B (en) | Report generation method and device based on data leakage prevention system | |
| CN117251358A (en) | Method, device, equipment and storage medium for testing application interface | |
| CN105788053A (en) | System and method for acquiring queuing and number calling data and data capture client | |
| CN114282265A (en) | Report sending method and device | |
| CN116126661A (en) | Method, equipment and medium for extracting and analyzing log of POS equipment | |
| CN115439057A (en) | Material management method, system and related device based on excel |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |