CN112866319B - Log data processing method, system and storage medium - Google Patents
Log data processing method, system and storage medium Download PDFInfo
- Publication number
- CN112866319B CN112866319B CN201911191616.0A CN201911191616A CN112866319B CN 112866319 B CN112866319 B CN 112866319B CN 201911191616 A CN201911191616 A CN 201911191616A CN 112866319 B CN112866319 B CN 112866319B
- Authority
- CN
- China
- Prior art keywords
- log data
- log
- shunting
- log file
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1044—Group management mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/24—Traffic characterised by specific attributes, e.g. priority or QoS
- H04L47/2441—Traffic characterised by specific attributes, e.g. priority or QoS relying on flow classification, e.g. using integrated services [IntServ]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L47/00—Traffic control in data switching networks
- H04L47/10—Flow control; Congestion control
- H04L47/29—Flow control; Congestion control using a combination of thresholds
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/30—Network architectures or network communication protocols for network security for supporting lawful interception, monitoring or retaining of communications or communication related information
- H04L63/306—Network architectures or network communication protocols for network security for supporting lawful interception, monitoring or retaining of communications or communication related information intercepting packet switched data communications, e.g. Web, Internet or IMS communications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/34—Network arrangements or protocols for supporting network services or applications involving the movement of software or configuration parameters
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Physics & Mathematics (AREA)
- Technology Law (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application relates to a log data processing method, a log data processing system and a storage medium. The method comprises the following steps: acquiring corresponding log data through more than one business servers corresponding to different service types; each service server generates corresponding log files from the log data according to a preset format; each service server sends the corresponding log files to the distribution processing cluster respectively; the shunting processing cluster shunts the log data according to the generation time of each log data in the log file to obtain a first shunting log file and a second shunting log file; the first split log file comprises full log data, and the second split log file comprises real-time log data; and the shunting processing cluster stores the first shunting log file and the second shunting log file to corresponding storage equipment respectively. By adopting the method, the log data processing efficiency can be improved.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a log data processing method, system, and storage medium.
Background
With the development of computer technology, service functions provided by various companies are more and more complex, corresponding system servers are more and more, and effective processing of log data plays a vital role in unified management and maintenance of various modules of various systems. In the conventional art, corresponding log data is also stored in different servers for different systems and different service functions. When the log data is required to be checked, all servers can be traversed in sequence to inquire the related log data.
However, the current log data processing method is complex in terms of log data query, rapid problem positioning and the like, so that the log data processing efficiency is low.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a log data processing method, system, and storage medium capable of improving log data processing efficiency.
A log data processing method, the method comprising:
acquiring corresponding log data through more than one business servers corresponding to different service types;
each service server generates the corresponding log file from the log data according to a preset format;
Each service server sends the corresponding log files to the distribution processing cluster respectively;
the shunting processing cluster shunts the log data according to the generation time of each log data in the log file to obtain a first shunting log file and a second shunting log file; the first shunt log file comprises full log data, and the second shunt log file comprises real-time log data;
and the shunting processing cluster stores the first shunting log file and the second shunting log file to corresponding storage equipment respectively.
A log data processing system comprising more than one service server corresponding to different service types, a distribution processing cluster and a storage device,
each service server is used for acquiring corresponding log data;
each service server is further used for generating a corresponding log file from the log data according to a preset format;
each service server is further used for respectively sending the corresponding log files to the distribution processing clusters;
the shunting processing cluster is used for shunting the log data according to the generation time of each log data in the log file to obtain a first shunting log file and a second shunting log file; the first shunt log file comprises full log data, and the second shunt log file comprises real-time log data;
And the shunting processing cluster is further used for respectively storing the first shunting log file and the second shunting log file to corresponding storage equipment.
A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
acquiring corresponding log data through more than one business servers corresponding to different service types;
each service server generates the corresponding log file from the log data according to a preset format;
each service server sends the corresponding log files to the distribution processing cluster respectively;
the shunting processing cluster shunts the log data according to the generation time of each log data in the log file to obtain a first shunting log file and a second shunting log file; the first shunt log file comprises full log data, and the second shunt log file comprises real-time log data;
and the shunting processing cluster stores the first shunting log file and the second shunting log file to corresponding storage equipment respectively.
According to the log data processing method, the system and the storage medium, the corresponding log data are acquired through more than one service servers corresponding to different service types, and the corresponding log data of the service servers of the service types have different log formats, so that the service servers can generate the corresponding log files according to the preset formats, and the log data in the log files have the same log format. The shunting processing cluster can shunt the log data according to the generation time of each log data in the log file to obtain a first shunting log file and a second shunting log file, and store the first shunting log file and the second shunting log file respectively, so that log inquiry operation is more convenient and faster in problem positioning, and further the log data processing efficiency is improved.
Drawings
FIG. 1 is an application scenario diagram of a log data processing method according to an embodiment;
FIG. 2 is a flowchart of a log data processing method according to an embodiment;
FIG. 3 is a schematic diagram of a log data acquisition process of a user terminal according to an embodiment;
FIG. 4 is a flow diagram of a log data offloading process in one embodiment;
FIG. 5 is a system association flow diagram of a log data processing method according to an embodiment;
fig. 6 is an internal structural diagram of a computer device in one embodiment.
Detailed Description
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.
The log data processing method provided by the application can be applied to an application environment shown in figure 1. The application environment includes more than one business server 102, a offload processing cluster 104, and a storage device 106, which correspond to different service types. Each service server 102 communicates with the distribution processing cluster 104 via a network, and the distribution processing cluster 104 communicates with each storage device 106 via a network. Each service server 102 may be implemented as an independent server or a server cluster formed by a plurality of servers. Each storage device 106 may be a terminal or a server. The terminal can be a desktop terminal or a mobile terminal, and the mobile terminal can be at least one of a mobile phone, a tablet computer, a notebook computer and the like.
Each service server 102 obtains log data corresponding to each service server; each service server 102 generates a corresponding log file from the log data according to a preset format; each service server 102 sends the log files corresponding to each service server to the distribution processing cluster; the shunting processing cluster 104 shunts the log data according to the generation time of each log data in the log file to obtain a first shunting log file and a second shunting log file; the split processing cluster 104 stores the first split log file and the second split log file, respectively, to corresponding storage devices 106. Those skilled in the art will appreciate that the application environment shown in fig. 1 is only a partial scenario related to the present application, and does not constitute a limitation on the application environment of the present application.
In one embodiment, as shown in fig. 2, a log data processing method is provided, which is applied to each service server 102, the shunting processing cluster 104 and the storage device 106 in fig. 1, and is illustrated as an example, and includes the following steps:
s202, acquiring the corresponding log data through more than one business servers corresponding to different service types.
The service servers are servers with corresponding service logic processing functions, and the service servers with different service types can be universal servers and micro servers. The general server is a server having a function of processing a user terminal and a local service logic of the general server. The micro server is a server with a service logic function corresponding to the processing micro service. The log data is information for recording hardware, software and system problems in the system, and can monitor events occurring in the system at the same time, and is used for checking the reasons of errors or searching traces left by an attacker when the system is attacked.
Specifically, more than one service server corresponding to different service types is deployed in the log data processing system, and log data corresponding to each service server can be obtained through each service server.
In one embodiment, the service servers of different service types include a general server and a micro server, and step S202, that is, the step of obtaining log data corresponding to each service server through more than one service servers of different service types, specifically includes: acquiring log data locally generated by a user terminal and a universal server through the universal server; and acquiring log data locally generated by the micro server through the micro server.
Specifically, the user operates a front page of the user terminal, and the user terminal may generate corresponding front log data. The application layer service and the management background service in the general server may generate log data when each provides a corresponding service. The log data generated locally by the user terminal and the universal server can be obtained through the universal server. The micro server can generate log data in the process of providing the micro service, and the log data generated locally by the micro server is obtained through the micro server. In this way, log data can be obtained quickly.
In one embodiment, the step of obtaining, by the general server, log data locally generated by the user terminal and the general server specifically includes: the universal server receives log data sent by different user terminals; the method comprises the steps that a universal server obtains log data locally generated by the universal server; the step of the user terminal sending the log data comprises: the user terminal acquires log data generated by the user terminal through a front-end page frame which is fused with a log data interceptor in advance; the user terminal asynchronously transmits the log data generated by the user terminal to the universal server.
The log data interceptor is an interceptor for intercepting all normal and abnormal log data and is used for acquiring log data generated when a user operates a front page of the user terminal. The front-end page frame is a frame used by a developer in developing a website page, and specifically may be a Bootstrap web page frame), amazeUI (Amaze User Interface, cross-screen front-end frame), layUI (Lay User Interface, front-end frame), and the like.
Specifically, the front end page frame is pre-fused with the log data interceptor, when a user operates the user terminal, the user terminal can acquire the log data generated by the user terminal through the front end page frame pre-fused with the log data interceptor. The user terminal can asynchronously transmit the log data generated by the user operating at the user terminal to the corresponding universal server, and the universal server can receive the log data sent from different user terminals. The universal server can generate corresponding log data when providing corresponding service, and the universal server can directly acquire the locally generated log data.
In one embodiment, as shown in fig. 3, as shown by a log data obtaining flow diagram of a user terminal, a user can access a page through the user terminal, the page can load resources corresponding to user operations from the internet, and when the loading of the resources by the page is completed, the user terminal can obtain all normal and abnormal log data generated by the user terminal through pre-fusing the monitoring browser events in the front end frame. The user terminal can transmit the log data generated by the user terminal to the corresponding server through asynchronous transmission.
In one embodiment, a common exception interceptor may be added to the front-end page frame, and log data of exceptions generated when a user operates the user terminal may be obtained by fusing the common exception interceptor to the front-end page frame in advance. The exception interceptor may invoke the common code to access the log data of the corresponding server save exception. Thus, all normal and abnormal log data corresponding to each service server can be obtained more quickly.
S204, each service server generates corresponding log files from the log data according to a preset format.
Specifically, the log data from different sources have different log configuration files, and each service server can generate the log file corresponding to the log data with the unified format according to the preset format.
In one embodiment, the Log file may be configured by Log4j (Log 4j—log for JAVA, log of JAVA). Log4j is a Log component that can be configured in Log format, log4j includes three important components: a log logger, an output and a log formatter. The logger is used to control which log statements are to be enabled or disabled and to place a level restriction on the log information. The output is used to specify whether the log is to be printed into the console or a file. The log formatter is used for controlling the display format of the log information.
In one embodiment, the log format may include different log fields, which may include, in particular, date, time, log level, code location, log content, error code, and the like. Each service server can carry out custom setting on the sequence of the log fields according to service requirements. The present application is not limited to log fields herein.
S206, each business server sends the corresponding log files to the distribution processing clusters.
The shunting processing cluster is a cluster with a shunting processing function for log data. Specifically, each service server stores a log file corresponding to each service server, and each service server may send the log file corresponding to each service server to the offloading processing cluster.
In one embodiment, step S206, that is, the step of each service server sending the log file corresponding to each service server to the splitting processing cluster, specifically includes: each service server monitors the corresponding log files through the acquisition codes; when each business server monitors the update of the corresponding log file, the updated log files corresponding to each business server are respectively sent to the distribution processing cluster.
Wherein the acquisition code is a code having a function of monitoring and acquiring log data from the corresponding log file. Specifically, each service server is deployed with a corresponding acquisition code, and the log file can update log data in the log file with the increase of time. Each business server can monitor the corresponding log file through the acquisition codes. When each service server monitors the update of the corresponding log file, each service server can respectively send the updated log file corresponding to each service server to the distribution processing cluster.
In one embodiment, the acquisition code may be specifically a code written in JAVA (object oriented programming language), and the acquisition code may be deployed on each service server and the path monitored by the acquisition code is specified. It can be understood that all the functional codes running on each service server can record the log data corresponding to each service server and output the recorded log data to the corresponding log file. The acquisition code can monitor all log files, and when the corresponding log files are monitored, log data reading is carried out. The acquisition code may record the current read location. When the next log file update is monitored, the log data continues to be read from the last recorded location. Therefore, each service server can send the latest log data to the shunting processing cluster, and the log processing efficiency is further improved.
S208, shunting processing is carried out on the log data according to the generation time of each log data in the log file by the shunting processing cluster, so as to obtain a first shunting log file and a second shunting log file; the first split log file includes a full volume of log data and the second split log file includes real-time log data.
The first split log file is a history log file, and is used for recording all log data corresponding to each service server, for example, the log data before 6 months can be queried through the first split log file. The first shunting log file comprises a total amount of log data, wherein the total amount of log data is all log data screened from all log data of the corresponding log file according to a preset log field. The second shunt log file is a real-time log file in a preset time period, and is used for recording log data in the preset time period corresponding to each service server, for example, the preset time period is 15 days, and only log data before 15 days can be queried through the second shunt log file. The second shunt log file comprises real-time log data, wherein the real-time log data is selected from all log data of the log file according to a preset time period. Wherein the full log data comprises real-time log data.
Specifically, a shunting processing cluster is deployed in the log data processing system, and each log data in the log file comprises the generation time of each log data. The shunting processing cluster can carry out shunting processing on the log data according to the generation time of each log data in the log file to obtain a first shunting log file and a second shunting log file.
In one embodiment, the splitting processing cluster performs splitting processing on the log data, specifically, a splitting time threshold may be set according to the generation time of each log data in the log file, and then the splitting processing is performed on the log data according to the splitting time threshold. For example, the split time threshold is 15 days, and the split processing cluster may generate the first split log file based on all log data within 15 days and outside 15 days. The split processing cluster may generate a second split log file based on log data within approximately 15 days.
In one embodiment, the offloading cluster may receive log data sent from each service server through KAFKA (distributed publish-subscribe messaging system), and may offload the log data through file (distributed stream data flow engine). As shown in fig. 4, the file automatically consumes the LOG data in the KAFKA, and after the LOG data is read from the KAFKA, the file cleans and FILTERs the LOG data through an acsp_log_source component, a trim_json component, a null_filter component, an acsp_log_bdp_format component, an acsp_log_hive component, and an acsp_log_es (SINK) component. It is to be appreciated that the ACSP_LOG_SOURCE component can consume data from the KAFKA's ESG_ACSP_CORE_BUS_LOG' theme. The trim_json component may format each row of log consumed from KAFKA to obtain the data content in JSON format therein, and convert the data content into a JSON string by JSON tool class. The null_filter component intercepts data-empty content generated during the formatting process and does not pass downstream. The acsp_log_bdp_format component splits JSON data content, obtains a preset number of fields, such as five fields, and stores the number of fields in an HIVE (data warehouse analysis system) database. The ACSP LOG ES component is a third party ES (distributed full text search) search platform for storing LOG data and supporting fast searches.
S210, the shunting processing cluster stores the first shunting log file and the second shunting log file to corresponding storage equipment respectively.
Specifically, the stream processing cluster may obtain addresses of storage devices corresponding to the first split log file and the second split log file, respectively. The splitting processing cluster may store the first splitting log file and the second splitting log file to respective storage devices according to addresses of the respective storage devices.
In one embodiment, the storage device may be specifically HIVE and ES retrieval engines. Specifically, the split processing cluster may store the first split log file in HIVE physical storage, facilitating subsequent full log data statistics. For example, a page may be made in the management background service, through which log data at any time may be queried, for example, log data 3 months ago may be queried. The offload processing cluster may store the second offload log file in the ES retrieval engine, and the offload processing cluster may set a time period for which the second offload log file is stored in the ES retrieval engine. For example, the second split log file can only be stored in the ES search engine for 15 days, and log data after 15 days will be automatically deleted. A page can be made in the management background service, and real-time log data of 15 days at present can be queried through the ES retrieval engine.
In one embodiment, as shown in fig. 5, a client accesses a user terminal in an extranet application through the internet, the user terminal may present a front-end page, and the client operates the front-end page of the user terminal to generate log data. The user terminal may asynchronously transfer the log data to a JETTY (lightweight container) application layer. Meanwhile, the user terminal and the application layer can also call normal service. The application layer can communicate with the DUBBOX (open source micro service architecture) micro service through a PUBLIC gateway, and the application layer can communicate with the management background service through a domain name address and asynchronously transmit log data generated by the user terminal and the application layer to the management background service. The management background service can store Log data generated by the service, received Log data of the user terminal and the application layer into Log4j Log files of the server. The Log generated by the micro-service may be recorded in Log4j Log files of the micro-service itself. And collecting programs are respectively deployed on the management background service and the micro service, and can regularly monitor and collect all log files on the management background service and the micro service and respectively send the collected log files, namely BEE (archive) files, to the KAFKA cluster. FLINK can automatically consume the log data of the log files in KAFKA, and shunt all log data. FLINK can store all log data in HIVE physical storage in full quantity, so that subsequent full quantity statistics is facilitated. FLINK may store log data for approximately 15 days in the ES retrieval engine. A query display page can be made in the management background to query real-time log data in the ES and historical log data in the HIVE.
In one embodiment, the front end log data generated by the user terminal, the log data generated by the application layer, the log data generated by the background management service, and the log data generated by the micro service have different log formats. The corresponding service servers can generate log files with the same log format according to a preset format. The corresponding service servers may send all log files with the same log format to the KAFKA cluster, respectively. The KAFKA can collect and sample the log data in all the received log files, and check whether the log format corresponding to the log data meets the service requirement. When the KAFKA log collection samples do not meet the service requirements, the preset format can be reset, and a corresponding log file which does not meet the service requirements is generated.
In the above log data processing method, the corresponding log data are obtained through more than one service servers corresponding to different service types, and as the corresponding log data of the service servers of the service types have different log formats, the corresponding log files can be generated by the service servers according to the preset formats, so that the log data in the log files have the same log format. The shunting processing cluster can shunt the log data according to the generation time of each log data in the log file to obtain a first shunting log file and a second shunting log file, and store the first shunting log file and the second shunting log file respectively, so that log inquiry operation is more convenient and faster in problem positioning, and further the log data processing efficiency is improved.
In one embodiment, step S204, that is, the step of generating, by each service server, the log data into the corresponding log file according to the preset format, specifically includes: each service server determines the same preset format according to preset service requirements; and each service server generates the corresponding log file from the log data according to the preset fields in the same preset format and the sequence of the preset fields.
Specifically, the log data obtained by each service server has different log formats, and each service server can determine the same preset format according to preset service requirements. The preset format may include preset fields and an ordering of the preset fields. Each service server can generate the corresponding log file from the log data according to the preset fields and the sequences of the preset fields in the same preset format.
In one embodiment, the preset field includes Log levels, and Log data information to be output in Log4j defines 6 Log levels, which are TRACE, DEBUG, INFO, WARN, ERROR and fault in sequence. When outputting the log, only log data information having a level higher than that specified in the configuration can be truly outputted. Wherein TRACE is a very low log level, it tracks the call of functions, and TRACE cannot contain variable parameters, but only prompts the call relationship of functions. DEBUGs are fine-grained levels that are used primarily to print some running information during development. INFO is a coarse level of granularity that highlights the running process of an application, printing some information of interest or importance to the user. WARN indicates that a potential error may occur, some of which are not error messages, but also provide some hints to the developer. ERROR indicates that, despite ERROR events, continued operation of the system is not affected for printing ERROR and exception information. The FATAL is a high log level that indicates that each serious error event will cause an exit of the application.
In the above embodiment, the unified log format is set through the preset field, so that it is more convenient to manage log data according to service requirements, and it is convenient to query the log data in the same page in a unified way.
In one embodiment, step S208, that is, a step of splitting log data according to generation time of each log data in the log file by the splitting processing cluster, to obtain a first splitting log file and a second splitting log file, specifically includes: the shunting processing cluster determines the generation time of each log data in the log file according to the time field in the preset format; the shunting processing cluster performs descending order sorting on the generation time of each log data in the log file to obtain a corresponding time sorting result; the shunting processing cluster screens out first shunting log data conforming to a first preset condition from all log data of the log file according to a time sequencing result and a preset field to be stored, and generates a first shunting log file according to the first shunting log data; and screening second shunt log data meeting a second preset condition from all log data of the log file according to a time sequencing result by the shunt processing cluster, and generating a second shunt log file according to the second shunt log data.
Specifically, the preset format includes a time field, and each piece of log data corresponds to the generation time of the log data. The shunting processing cluster can determine the generation time of each log data in the log file according to the time field in the preset format. The shunting processing cluster can sort the generation time of each log data in the log file in a descending order to obtain a corresponding time sorting result. The distribution processing cluster can determine a preset field to be stored, a first preset condition and a second preset condition corresponding to the log data according to the service requirement. The shunting processing cluster can screen out first shunting log data meeting a first preset condition from all log data of the log file according to a time sequencing result and a preset field to be stored, and generate a first shunting log file according to the first shunting log data. The shunting processing cluster can screen out second shunting log data meeting a second preset condition from all log data of the log file according to a time sequencing result, and generate the second shunting log file according to the second shunting log data.
In one embodiment, the first preset condition may specifically be that the log data corresponding to the 5 fields are screened from all log data corresponding to the time sequencing result to perform full-scale storage, for example, the 5 fields may specifically be timeMillis, thread, level, log name and message. The second preset condition may specifically be that, from all log data corresponding to the time sequencing result, log data corresponding to the last 15 days is screened and stored in real time.
In the above embodiment, the log data is split according to the first preset condition and the second preset condition, so that the log data can be classified and managed according to the service requirement, and the log data processing efficiency is further improved.
In one embodiment, the storage device includes a full-volume storage device and a real-time storage device, and step S210, that is, the step of storing, by the splitting processing cluster, the first splitting log file and the second splitting log file to the corresponding storage devices, includes: the shunting processing cluster respectively acquires the address of the full-capacity storage device and the address of the real-time storage device; the shunting processing cluster stores the first shunting log file into a physical memory of the full-capacity storage device according to the address of the full-capacity storage device; and the shunting processing cluster stores the second shunting log file to the real-time storage device according to the address of the real-time storage device.
The full-volume storage device is a device for storing the first split-flow log file in a full-volume mode, and the real-time storage device is a device for storing the second split-flow log file in a real-time mode. Specifically, the log data processing system is provided with a full-capacity storage device and a real-time storage device, and the shunting processing cluster can respectively acquire the address of the full-capacity storage device and the address of the real-time storage device. The split processing cluster may find the full-size storage device according to an address of the full-size storage device, and further store the first split log file into a physical memory of the full-size storage device. The shunting processing cluster can find out the real-time storage device according to the address of the real-time storage device, and store the second storage shunting log file into the real-time storage device.
In one embodiment, a developer may add a tracking identifier of a user terminal request to a function code corresponding to the user terminal, and then, according to the tracking identifier, may correlate log data generated by the user terminal with log data generated by each service server. A log data query page is generated in the management background service, through which the full log data in the full storage device can be queried in a link manner, and through which the real-time log data in the real-time storage device can be queried in a link manner.
In the above embodiment, according to the address of the full-capacity storage device and the address of the real-time storage device, the splitting processing cluster may quickly query the full-capacity storage device and the real-time storage device, so as to update and store the first splitting log file and the second splitting log file in the corresponding storage devices in real time, thereby improving the storage efficiency of the log data.
In a specific embodiment, the log data processing method includes the steps of:
the universal server receives log data sent by different user terminals.
The user terminal acquires the log data generated by the user terminal through a front-end page frame which is fused with a log data interceptor in advance.
The user terminal asynchronously transmits the log data generated by the user terminal to the universal server.
The universal server obtains log data generated locally by the universal server.
And acquiring log data locally generated by the micro server through the micro server.
And each service server determines the same preset format according to the preset service requirement.
And each service server generates the corresponding log file from the log data according to the preset fields in the same preset format and the sequence of the preset fields.
And each business server monitors the corresponding log file through the acquisition codes.
When each business server monitors the update of the corresponding log file, the updated log files corresponding to each business server are respectively sent to the distribution processing cluster.
And determining the generation time of each log data in the log file by the shunting processing cluster according to the time field in the preset format.
And the shunting processing cluster performs descending order sequencing on the generation time of each log data in the log file to obtain a corresponding time sequencing result.
And screening first shunting log data meeting a first preset condition from all log data of the log file according to a time sequencing result and a preset field to be stored by the shunting processing cluster, and generating a first shunting log file according to the first shunting log data.
And screening second shunt log data meeting a second preset condition from all log data of the log file according to a time sequencing result by the shunt processing cluster, and generating a second shunt log file according to the second shunt log data.
The shunting processing cluster respectively acquires the address of the full-capacity storage device and the address of the real-time storage device.
And the shunting processing cluster stores the first shunting log file into a physical memory of the full-capacity storage device according to the address of the full-capacity storage device.
And the shunting processing cluster stores the second shunting log file to the real-time storage device according to the address of the real-time storage device.
According to the log data processing method, the corresponding log data are acquired through more than one service servers corresponding to different service types, and the corresponding log data of the service servers of the service types have different log formats, so that the corresponding log files can be generated by the service servers according to the preset formats, and the log data in the log files have the same log format. The shunting processing cluster can shunt the log data according to the generation time of each log data in the log file to obtain a first shunting log file and a second shunting log file, and store the first shunting log file and the second shunting log file respectively, so that log inquiry operation is more convenient and faster in problem positioning, and further the log data processing efficiency is improved.
It should be understood that, although the steps in the above specific embodiment are shown sequentially in order, the steps are not necessarily performed sequentially in order. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the specific embodiments described above may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of execution of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with at least a portion of other steps or other steps.
In one embodiment, referring to FIG. 1, there is provided a log data processing system comprising: more than one traffic server 102, corresponding to different service types, a offload processing cluster 104, and a storage device 106, wherein:
each service server 102 is configured to obtain log data corresponding to each service server.
Each service server 102 is further configured to generate a corresponding log file from the log data according to a preset format.
Each service server 102 is further configured to send a corresponding log file to the splitting processing cluster 104.
The splitting processing cluster 104 is configured to split the log data according to the generation time of each log data in the log file, so as to obtain a first split log file and a second split log file; the first split log file includes a full volume of log data and the second split log file includes real-time log data.
The shunting processing cluster 104 is further configured to store the first shunting log file and the second shunting log file to corresponding storage devices 106, respectively.
In one embodiment, each service server 102 is further configured to obtain log data locally generated by the user terminal and the universal server; log data generated locally by the micro server is obtained.
In one embodiment, each service server 102 is further configured to receive log data sent by different user terminals; acquiring log data locally generated by a universal server; the step of the user terminal sending the log data comprises: the user terminal acquires log data generated by the user terminal through a front-end page frame which is fused with a log data interceptor in advance; the user terminal asynchronously transmits the log data generated by the user terminal to the universal server.
In one embodiment, each service server 102 is further configured to determine the same preset format according to a preset service requirement; and generating the corresponding log file from the log data according to the preset fields in the same preset format and the sequence of the preset fields.
In one embodiment, each service server 102 is further configured to monitor a corresponding log file by collecting codes; when each service server 102 monitors the update of the corresponding log file, the updated log file corresponding to each service server is sent to the distribution processing cluster.
In one embodiment, the shunting processing cluster 104 is further configured to determine, according to a time field in a preset format, a generation time of each log data in the log file; the generation time of each log data in the log file is ordered in a descending order to obtain a corresponding time ordering result; screening first shunting log data meeting a first preset condition from all log data of the log file according to a time sequencing result and a preset field to be stored, and generating a first shunting log file according to the first shunting log data; and screening second shunt log data meeting a second preset condition from all log data of the log file according to the time sequencing result, and generating the second shunt log file according to the second shunt log data.
In one embodiment, the shunting processing cluster 104 is further configured to obtain an address of the full-capacity storage device and an address of the real-time storage device, respectively; storing the first shunting log file into a physical memory of the full-capacity storage device according to the address of the full-capacity storage device; and storing the second shunt log file to the real-time storage device according to the address of the real-time storage device.
According to the log data processing system, the corresponding log data are acquired through more than one service servers corresponding to different service types, and the corresponding log data of the service servers of the service types have different log formats, so that the corresponding log files can be generated by the service servers according to the preset formats, and the log data in the log files have the same log format. The shunting processing cluster can shunt the log data according to the generation time of each log data in the log file to obtain a first shunting log file and a second shunting log file, and store the first shunting log file and the second shunting log file respectively, so that log inquiry operation is more convenient and faster in problem positioning, and further the log data processing efficiency is improved.
In one embodiment, a computer device is provided, which may be each service server 102 in fig. 1, and the internal structure diagram thereof may be shown in fig. 6. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is for storing log data processing data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a log data processing method.
It will be appreciated by those skilled in the art that the structure shown in FIG. 6 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
In one embodiment, a computer device is provided that includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the log data processing method described above. The steps of the log data processing method here may be the steps in the log data processing method of each of the above embodiments.
In one embodiment, a computer readable storage medium is provided, storing a computer program which, when executed by a processor, causes the processor to perform the steps of the log data processing method described above. The steps of the log data processing method here may be the steps in the log data processing method of each of the above embodiments.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the application, which are described in detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (10)
1. A method of log data processing, the method comprising:
acquiring corresponding log data through more than one business servers corresponding to different service types;
each service server generates the corresponding log file from the log data according to a preset format;
each service server sends the corresponding log files to the distribution processing cluster respectively;
The shunting processing cluster shunts the log data according to the generation time of each log data in the log file to obtain a first shunting log file and a second shunting log file; the first shunt log file comprises full log data, and the second shunt log file comprises real-time log data; the full log data includes the real-time log data;
the shunting processing cluster respectively acquires the address of the full-capacity storage device and the address of the real-time storage device; the full-volume storage device is a storage device for storing full-volume log data, and the real-time storage device is a storage device for storing real-time log data;
the split processing cluster stores the first split log file to the full-capacity storage device according to the address of the full-capacity storage device;
and the shunting processing cluster stores the second shunting log file to the real-time storage device according to the address of the real-time storage device.
2. The method according to claim 1, wherein the service servers of different service types include a general server and a micro server, and the acquiring the log data corresponding to each service server by more than one service servers of different service types includes:
Acquiring log data locally generated by a user terminal and the universal server through the universal server;
and acquiring log data locally generated by the micro server through the micro server.
3. The method according to claim 2, wherein the obtaining, by the general server, log data generated locally by a user terminal and the general server, comprises:
the universal server receives log data sent by different user terminals;
the universal server acquires log data locally generated by the universal server;
the step of the user terminal sending log data comprises:
the user terminal acquires log data generated by the user terminal through a front-end page frame which is fused with a log data interceptor in advance;
and the user terminal asynchronously transmits the log data generated by the user terminal to the universal server.
4. The method of claim 1, wherein the generating, by each service server, the log data into a corresponding log file according to a preset format, includes:
each service server determines the same preset format according to preset service requirements;
And each service server generates the corresponding log file from the log data according to the preset fields and the sequences of the preset fields in the same preset format.
5. The method according to claim 1, wherein each of the service servers sends the corresponding log file to the distribution processing cluster, respectively, comprising:
each service server monitors the corresponding log file through the acquisition codes;
when each service server monitors the update of the corresponding log file, the updated log files corresponding to the service servers are respectively sent to the distribution processing cluster.
6. The method of claim 1, wherein the splitting cluster performs splitting on the log data according to the generation time of each log data in the log file to obtain a first split log file and a second split log file, and the method comprises:
the shunting processing cluster determines the generation time of each log data in the log file according to the time field in the preset format;
the shunting processing cluster performs descending order sequencing on the generation time of each log data in the log file to obtain a corresponding time sequencing result;
The shunting processing cluster screens out first shunting log data conforming to a first preset condition from all log data of the log file according to the time sequencing result and a preset field to be stored, and generates a first shunting log file according to the first shunting log data;
and the shunting processing cluster screens out second shunting log data meeting a second preset condition from all log data of the log file according to the time sequencing result, and generates a second shunting log file according to the second shunting log data.
7. A log data processing system is characterized in that the system comprises more than one business server corresponding to different service types, a distribution processing cluster, a full-quantity storage device and a real-time storage device,
each service server is used for acquiring corresponding log data;
each service server is further used for generating a corresponding log file from the log data according to a preset format;
each service server is further used for respectively sending the corresponding log files to the distribution processing clusters;
the shunting processing cluster is used for shunting the log data according to the generation time of each log data in the log file to obtain a first shunting log file and a second shunting log file; the first shunt log file comprises full log data, and the second shunt log file comprises real-time log data;
The distribution processing cluster is also used for respectively acquiring the address of the full-capacity storage device and the address of the real-time storage device; the full-volume storage device is a storage device for storing full-volume log data, and the real-time storage device is a storage device for storing real-time log data; storing the first split log file to the full storage device according to the address of the full storage device; and storing the second shunt log file to the real-time storage device according to the address of the real-time storage device.
8. The system of claim 7, wherein each service server comprises a generic server and a micro server; the universal server is used for acquiring log data locally generated by the user terminal and the universal server; the micro server is used for acquiring log data locally generated by the micro server.
9. The system of claim 7, wherein each service server is further configured to determine the same preset format according to a preset service requirement; and generating the corresponding log file from the log data according to the preset fields in the same preset format and the sequence of the preset fields.
10. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911191616.0A CN112866319B (en) | 2019-11-28 | 2019-11-28 | Log data processing method, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911191616.0A CN112866319B (en) | 2019-11-28 | 2019-11-28 | Log data processing method, system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112866319A CN112866319A (en) | 2021-05-28 |
| CN112866319B true CN112866319B (en) | 2023-10-13 |
Family
ID=75995596
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911191616.0A Active CN112866319B (en) | 2019-11-28 | 2019-11-28 | Log data processing method, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112866319B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114168672B (en) * | 2021-12-13 | 2022-09-23 | 明觉科技（北京）有限公司 | Log data processing method, device, system and medium |
| CN114661682A (en) * | 2021-12-24 | 2022-06-24 | 奇安信科技集团股份有限公司 | Log file generation method and device |
| CN114153823B (en) * | 2022-02-09 | 2022-05-17 | 北京华品博睿网络技术有限公司 | Distributed computing job log data processing method and system |
| CN117555874B (en) * | 2024-01-11 | 2024-03-29 | 成都大成均图科技有限公司 | Log storage method, device, equipment and medium of distributed database |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108599992A (en) * | 2018-03-21 | 2018-09-28 | 四川斐讯信息技术有限公司 | A kind of data processing system and method |
| CN109274540A (en) * | 2018-11-16 | 2019-01-25 | 四川长虹电器股份有限公司 | A kind of web access log processing method based on storm |
| CN109933505A (en) * | 2019-03-14 | 2019-06-25 | 深圳市珍爱捷云信息技术有限公司 | Log processing method, device, computer equipment and storage medium |
| CN110008104A (en) * | 2019-04-10 | 2019-07-12 | 苏州浪潮智能科技有限公司 | A kind of management method of log information, system, equipment and storage medium |
| CN110019008A (en) * | 2017-11-03 | 2019-07-16 | 北京金山安全软件有限公司 | Data storage method and device |
| US10372595B1 (en) * | 2016-12-15 | 2019-08-06 | EMC IP Holding Company LLC | System and method to diagnose applications by accessing log files associated with different subsystems of a data center via a common interface |
| CN110413478A (en) * | 2019-06-28 | 2019-11-05 | 苏州浪潮智能科技有限公司 | A kind of method, equipment and medium monitoring log processing |
-
2019
- 2019-11-28 CN CN201911191616.0A patent/CN112866319B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10372595B1 (en) * | 2016-12-15 | 2019-08-06 | EMC IP Holding Company LLC | System and method to diagnose applications by accessing log files associated with different subsystems of a data center via a common interface |
| CN110019008A (en) * | 2017-11-03 | 2019-07-16 | 北京金山安全软件有限公司 | Data storage method and device |
| CN108599992A (en) * | 2018-03-21 | 2018-09-28 | 四川斐讯信息技术有限公司 | A kind of data processing system and method |
| CN109274540A (en) * | 2018-11-16 | 2019-01-25 | 四川长虹电器股份有限公司 | A kind of web access log processing method based on storm |
| CN109933505A (en) * | 2019-03-14 | 2019-06-25 | 深圳市珍爱捷云信息技术有限公司 | Log processing method, device, computer equipment and storage medium |
| CN110008104A (en) * | 2019-04-10 | 2019-07-12 | 苏州浪潮智能科技有限公司 | A kind of management method of log information, system, equipment and storage medium |
| CN110413478A (en) * | 2019-06-28 | 2019-11-05 | 苏州浪潮智能科技有限公司 | A kind of method, equipment and medium monitoring log processing |
Non-Patent Citations (1)
| Title |
|---|
| 侯婧媖.信息系统平台日志监控报警工具的研制.《大众用电》.2015, * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112866319A (en) | 2021-05-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112866319B (en) | Log data processing method, system and storage medium | |
| CN112910945B (en) | Request link tracking method and service request processing method | |
| CN109522287B (en) | Monitoring method, system, equipment and medium for distributed file storage cluster | |
| CN110399293B (en) | System test method, device, computer equipment and storage medium | |
| CN108197200B (en) | Log tracking method and device, computer equipment and storage medium | |
| CN108804618B (en) | Database configuration method, device, computer equipment and storage medium | |
| US10771306B2 (en) | Log monitoring system | |
| CN110162512B (en) | Log retrieval method, device and storage medium | |
| CN109325010B (en) | Log checking method, device, computer equipment and storage medium | |
| CN110309109B (en) | Data monitoring method, device, computer equipment and storage medium | |
| CN113687974B (en) | Client log processing method and device and computer equipment | |
| CN105743730A (en) | Method and system used for providing real-time monitoring for webpage service of mobile terminal | |
| CN114595201A (en) | Method, equipment and storage medium for inquiring acquisition record of interface access log | |
| CN113220540B (en) | Service management method, device, computer equipment and storage medium | |
| CN110932933A (en) | Network condition monitoring method, computing device and computer storage medium | |
| CN113419950A (en) | Method and device for generating UI automation script, computer equipment and storage medium | |
| CN118409926A (en) | Log collection method, device, computer equipment and storage medium | |
| CN116644250B (en) | Page detection method, page detection device, computer equipment and storage medium | |
| CN114238085A (en) | Interface testing method and device, computer equipment and storage medium | |
| CN114528201A (en) | Abnormal code positioning method, device, equipment and medium | |
| CN113342806A (en) | Big data processing method and device, storage medium and processor | |
| CN112651840B (en) | Business data log processing method and system based on blockchain and digital finance | |
| CN114185798A (en) | Interface test case detection method and device, computer equipment and storage medium | |
| US9240968B1 (en) | Autogenerated email summarization process | |
| CN112765100A (en) | Method, system, computing device and medium for querying logs |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |