CN115357689B - Data processing method, device and medium of distributed log and computer equipment - Google Patents
Data processing method, device and medium of distributed log and computer equipment Download PDFInfo
- Publication number
- CN115357689B CN115357689B CN202211269620.6A CN202211269620A CN115357689B CN 115357689 B CN115357689 B CN 115357689B CN 202211269620 A CN202211269620 A CN 202211269620A CN 115357689 B CN115357689 B CN 115357689B
- Authority
- CN
- China
- Prior art keywords
- log
- log file
- user
- acquiring
- query
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3055—Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/338—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Abstract
The embodiment of the invention discloses a data processing method, a data processing device, a data processing medium and computer equipment of a distributed log. The method comprises the following steps: acquiring a distributed log file, adding an ID (identity) to the distributed log file and acquiring the log file with the ID; the log files with the IDs are sorted according to the ID identifications and then stored in a server of a log query platform; acquiring a query instruction, and acquiring a result log file from a server according to a query condition of the query instruction; and displaying the result log file on a Web terminal of the log query platform. And sequencing and storing the distributed log files according to the ID identification, so that when the log files are queried subsequently, the log files are data subjected to sequencing preprocessing, the query result is strictly consistent with the sequencing sequence, and the efficiency and the habit of the distributed log files are kept the same as those of the traditional terminal by displaying through the Web terminal.
Description
Technical Field
The present invention relates to the field of distributed log technologies, and in particular, to a data processing method, an apparatus, a medium, and a computer device for a distributed log.
Background
In the prior art, in a distributed log system, conventional log files are collected and stored, so that the log files can be inquired, called and analyzed later. The existing distributed log system is built by using ELK open source software, and logs generated by all nodes of the distributed log system are collected through message queues and then stored. The ELK open source software comprises: elasticSearch, logstash and Kiabana three open source tools.
The data volume of the distributed log file is large, which causes the data volume of distributed log monitoring to be large, and meanwhile, the source tracing and positioning of the analysis result of the log file are difficult due to a distributed deployment mode.
Disclosure of Invention
In view of the foregoing, it is necessary to provide a data processing method, apparatus, medium, and computer device for distributed logs.
A data processing method of distributed logs is applied to a distributed log system, and comprises the following steps:
and acquiring a distributed log file, adding an ID (identity) to the distributed log file, and acquiring the log file with the ID.
And sorting the log files with the IDs according to the ID identifications and storing the log files in a server of a log query platform.
And acquiring a query instruction, and acquiring a result log file from the server according to a query condition of the query instruction.
And displaying the result log file on a Web terminal of the log query platform.
Further, monitoring a log message queue in a server of a log query platform according to a query condition of the query instruction;
defining a listening class MQLogListener of the log message queue, the listening class MQLogListener being used for listening log changes of the log message queue, two long arrays having a first array length being defined in the MQLogListener: the first long array and the second long array are used for circularly storing a result log file;
storing the result log file in a first long array or a second long array and detecting whether the length of the stored long array exceeds the length of the first array;
if the length of the first array is not exceeded, monitoring a log message queue by regularly polling and monitoring a query interface TailQuery in a text box of a display screen of the Web terminal, the polling conditions of the regular polling monitoring query interface TailQuery comprise page filtering conditions, a final log ID and a maximum log length;
when a timing polling monitoring query interface TailQuery receives a query instruction, according to the last log ID of a Web terminal display screen, adopting the starting point position of a binary bit length array to start querying a result log file from the starting point position of a long array, and simultaneously filtering the result log file by using keywords until the long array reaches the maximum length;
performing splicing operation on the monitored and inquired logs by using an array push method to obtain a first spliced log file;
screening the first spliced log file through a fixed-length array, defining the fixed-length array through a Web terminal, and deleting the logs exceeding the fixed-length array if the first spliced log file exceeds the size of the fixed-length array;
splicing the filtered first spliced log files by using an array join method to obtain a complete log body, and taking the complete log body as a result log file;
obtaining the result log grade, and marking the fonts of the classified log files into different colors by using a color style according to the result log grade;
displaying a log using a < pre > tag tape style in a text box of a Web terminal display screen;
if the time sequence query is selected on the page of the scroll text box of the Web terminal display screen, the log display is automatically set to the top for display by the setTimeout delay of 50 milliseconds and by using a scrollTop (0), and the classified log files subjected to color rendering can be displayed in sequence from far to near;
if the page of the scroll text box of the display screen of the Web terminal selects time reverse query, automatically scrolling the log display to the lowest by using scrollTop (1000000) with the setTimeout delay of 50 milliseconds, and selecting to display the color-rendered classification log file from near to far.
Further, the steps of acquiring a distributed log file, adding an ID identifier to the distributed log file, acquiring a log file with an ID, sorting the log file with the ID according to the ID identifier, storing the log file in a server of a log query platform, acquiring a query instruction, acquiring a result log file from the server according to a query condition of the query instruction, and displaying the result log file on a Web terminal of the log query platform include:
intercepting and generating a distributed log file at a gateway, and adding an ID (identity) in the distributed log file, wherein the ID comprises a time ID, a log identity ID and a log link ID;
sorting the log files with the IDs according to the time IDs and storing the log files in a memory message queue in a server of a log query platform;
acquiring a query instruction, analyzing user information of the query instruction, acquiring a log Identity (ID) matched with the user information, screening a distributed log file according to the acquired log Identity (ID) and storing the distributed log file in a user message queue of a server;
and acquiring a result log file from a user message queue of the server according to the query instruction.
Further, the steps of acquiring a distributed log file, adding an ID identifier to the distributed log file, acquiring a log file with an ID, sorting the log file with the ID according to the ID identifier, storing the log file in a server of a log query platform, acquiring a query instruction, acquiring a result log file from the server according to a query condition of the query instruction, and displaying the result log file on a Web terminal of the log query platform include:
intercepting and generating a distributed log file at a gateway;
adding ID identification in a distributed log file, wherein the ID identification comprises a time ID, a user identity ID, a machine ID, a link ID, a service ID and an analysis software link ID;
sorting the log files with the ID identifications according to the time IDs and storing the log files in a memory message queue in a server of a log query platform;
acquiring a monitoring instruction, analyzing user information of the monitoring instruction, acquiring an ID (identity) matched with the user information, screening a distributed log file according to the ID, acquiring a user log file related to the user information, and storing the user log file in a first user message queue of a server or a second user message queue of the user server;
acquiring a query instruction, and acquiring a user result log file from a user server according to a query condition of the query instruction, or acquiring a user result log file from the user server;
and displaying the user result log file on the terminal of the user.
Further, the step of acquiring the ID matching with the user information includes:
obtaining the category of ID identification, wherein the category of ID identification comprises: tenant ID, enterprise ID, operating user ID;
analyzing the user information, and acquiring the tenant identity, the enterprise identity and the operation user identity of the user information;
and obtaining the ID identification matched with the user information according to the type of the ID identification and the analysis result of the user information.
Further, if the user result log level is a debugging DEBUG level, the font of the classification log file is marked as a first color by using a color style;
if the user result log level is the warning WARN level, the font of the classification log file is marked to be a second color by using a color style;
if the user result log level is the ERROR ERROR level, the font of the classification log file is marked to be a third color by using a color style;
if the user result log level is a normal level, the font of the classification log file is a fourth color;
a log display using a < pre > tag tape style is displayed in a text box of a display screen of the user terminal.
Further, acquiring the color of the classification log file;
acquiring the color scrolling speed of the classification log file according to the color of the classification log file;
displaying the classification log file in a scrolling text box according to the color scrolling speed;
the scrolling speed of the log file of the first color is greater than that of the log file of the second color, the scrolling speed of the log file of the second color is greater than that of the log file of the third color, and the scrolling speed of the log file of the third color is greater than that of the log file of the fourth color;
and obtaining color type information of the user result log file, wherein the color type information comprises the proportion of the first color, the proportion of the second color, the proportion of the third color and the proportion of the fourth color, and obtaining a log analysis result according to the color type information of the user result log file.
Further, the steps of obtaining a monitoring instruction, analyzing the user information of the monitoring instruction, obtaining an ID identifier matched with the user information, screening the distributed log file according to the ID identifier, and obtaining a user log file related to the user information include:
acquiring a monitoring instruction, analyzing user information of the monitoring instruction, acquiring an ID (identity) identifier matched with the user information, wherein the ID identifier comprises an analysis software link ID, screening distributed log files according to the ID identifier and acquiring user log files related to the user information;
acquiring the operation/processing occupation time of a user log file by analyzing the software link ID;
acquiring the indentation width of the log file according to the operation/processing occupation time of the user log file, wherein the longer the operation/processing occupation time of the user log file is, the longer or the shorter the indentation width of the corresponding log file is;
and acquiring a performance analysis result of the user log file according to the indentation width corresponding to the user log file.
Further, the step of obtaining the monitoring instruction, analyzing the user information of the monitoring instruction, obtaining an ID identifier matched with the user information, screening the distributed log file according to the ID identifier and obtaining a user log file related to the user information, storing the user log file in a first user message queue of the server or in a second user message queue of the user server, obtaining the query instruction, obtaining a user result log file from the user server according to a query condition of the query instruction, or obtaining the user result log file from the user server includes:
acquiring a monitoring instruction, analyzing user information of the monitoring instruction, acquiring an ID (identity) identifier matched with the user information, screening a distributed log file according to a link ID, and acquiring a user log file according to the link ID;
screening the distributed log files according to the identity IDs, clustering the screening results through the link IDs and obtaining clustering result log files, wherein the method for clustering the screening results comprises the same-thread clustering, the same-machine clustering, the same-service clustering and the same-environment clustering;
copying and sending the clustering result log file to a user message queue of a user server corresponding to the monitoring instruction;
acquiring a query instruction, analyzing the query condition in the query instruction, acquiring a co-linear result log file or a co-machine clustering result log file or a co-service clustering result log file or a co-environment clustering result log file in a clustering result log file according to the query condition, and taking the co-linear result log file or the co-machine clustering result log file or the co-service clustering result log file or the co-environment clustering result log file as a user result log file;
and displaying the user result log file on the terminal of the user.
According to the data processing method of the distributed log, when the distributed log file is obtained, the ID identification is added in the distributed log file, the distributed log file is sorted and stored according to the ID identification, when the log file is inquired subsequently, the log file is data subjected to sorting pretreatment, the inquiry result is strictly consistent with the sorting sequence, the efficiency and the habit which are the same as those of a traditional terminal are kept through the display of a Web terminal, and meanwhile, better log data are provided for the realization of the functions of change monitoring, color rendering and automatic scrolling of the result log file subsequently.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Wherein:
FIG. 1 is a flow diagram that illustrates a method for data processing for a distributed journal, under an embodiment;
FIG. 2 is a flowchart of a data processing method of a distributed log in another embodiment;
FIG. 3 is a block diagram of a data processing apparatus for distributed logging in one embodiment;
FIG. 4 is a block diagram of a computer device in one embodiment.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
The embodiment provides a data processing method of a distributed log, which can be applied to a server of a distributed log system and also can be applied to a user terminal.
The technical problem to be solved by the scheme is that the existing distributed log system built through ELK open source software stores log files generated by each node after being gathered through message queues, the data volume of the generated distributed logs is very large due to the characteristics of the distributed log system, the monitoring data generated by monitoring tasks on the distributed logs is also very large, and the source tracing and positioning of the finally generated log files are disordered and complicated due to distributed deployment.
In order to solve the foregoing technical problem, this embodiment provides a data processing method for a distributed log, which is applied to a distributed log system, and in particular, in this embodiment, the method is applied to a server of a log query platform, as shown in fig. 1, the method includes the following steps:
s101: and acquiring a distributed log file, adding an ID (identity) to the distributed log file, and acquiring the log file with the ID.
And printing the interception log and adding the ID I into each subsystem of the distributed configuration of the distributed log system to form a log file with the ID, so that the log file can be conveniently processed according to the ID in the follow-up process.
The log file with the ID may be sent to a message queue.
S102: the log files with the IDs are sorted according to the ID identifications and then stored in a server of a log query platform; the log files with the IDs are sorted according to the ID identifications, specifically, the log files with the IDs can be sorted and stored through a log analysis tool logstack, and specifically, the log files can be stored in a log storage module ElasticSearch of a server of a log query platform.
S103: acquiring a query instruction, and acquiring a result log file from a server according to a query condition of the query instruction; the query instruction can be a query instruction sent by a user, and the query method of the query instruction comprises the following steps: inquiring a log file from a server of a log inquiry platform according to the ID through a Web polling inquiry method; or, the log file is inquired from a server of the log inquiry platform according to the ID through a keyword inquiry method.
S104: and displaying the result log file on a Web terminal of the log query platform.
The query result is displayed through the Web terminal of the log query platform, and because the log file is data subjected to sorting pretreatment, the query result is strictly consistent with the sorting sequence, the efficiency and the habit of the traditional terminal are kept through the Web terminal display, and better log data are provided for realizing the functions of subsequently performing change monitoring, color rendering and automatic screen scrolling on the result log file.
Example two
The embodiment provides a specific process of a data processing method of a specific distributed log, which comprises the following steps:
the method comprises the steps that log files with ID identifications generated by all nodes of a distributed log system are obtained by printing interception log files in a plurality of distributed log subsystems and adding ID identifications; sending the log file with the ID identification to a message queue;
the method comprises the steps that log files with ID identifications in message queues are sequenced through a log parsing tool logstack and stored in a log storage module ElasticSearch of a server of a log query platform, wherein the log storage module ElasticSearch comprises log message queues;
acquiring a query instruction, and monitoring a log message queue in a server of a log query platform according to a query condition of the query instruction;
defining a monitoring class MQLogListener of a log message queue, wherein the monitoring class MQLogListener is used for monitoring log changes of the log message queue, and two long arrays with the size of 1 ten thousand are defined in the MQLogListener: the first long array and the second long array are used for circularly storing a result log file;
storing the result log file in a first long array or a second long array and detecting whether the length of the stored long array exceeds 1 ten thousand;
if the number of the log messages is not more than 1 ten thousand, monitoring a log message queue in a text box of a display screen of the Web terminal by polling and monitoring an inquiry interface TailQuery at regular time, wherein the polling conditions of the inquiry interface TailQuery at regular time comprise a page filtering condition, a final log ID and a maximum log length;
when a timing polling monitoring query interface TailQuery receives a query instruction, according to the last log ID of a Web terminal display screen, adopting the starting point position of a binary bit length array to start querying a result log file from the starting point position of a long array, and simultaneously filtering the result log file by using keywords until the long array reaches the maximum length;
using an array push method to carry out splicing operation on the logs searched and monitored to obtain a first spliced log file;
screening the first spliced log file through a fixed-length array, defining the fixed-length array through a Web terminal, and deleting the logs exceeding the fixed-length array if the first spliced log file exceeds the size of the fixed-length array;
splicing the filtered first spliced log files by using an array join method to obtain a complete log body, and taking the complete log body as a result log file;
if the result log level is the debugging DEBUG level, the font of the classification log file is marked as gray #909399 by using a color style;
if the result log level is the warning WARN level, the font of the classification log file is marked as yellow # E6A23C by using a color style;
if the result log level is an ERROR ERROR level, the font of the classification log file is marked as red # F56C6C by using a color style;
if the result log level is a normal level, the font of the classification log file is black;
displaying a log in a text box of a Web terminal display screen by using a < pre > tag tape style;
if the time sequence query is selected on the page of the scroll text box of the Web terminal display screen, the log display is automatically set to the top for display by the setTimeout delay of 50 milliseconds and by using a scrollTop (0), and the classified log files subjected to color rendering can be displayed in sequence from far to near;
if the page of the scroll text box of the display screen of the Web terminal selects time reverse query, automatically scrolling the log display to the lowest by using scrollTop (1000000) with the setTimeout delay of 50 milliseconds, and selecting to display the color-rendered classification log file from near to far.
By the aid of the scheme, log files can be strictly sorted according to time, log changes of a log message queue are monitored through a monitoring MQLogListener, log splicing is carried out on monitoring results, a complete log body is formed, and the result log files are obtained. And conveniently tracing and positioning the generating node of the log change by the ID identification. The solution efficiency of the system problem is improved.
EXAMPLE III
The embodiment provides a data processing method of a distributed log, which is applied to a distributed log system and comprises the following steps:
intercepting and generating a distributed log file at a gateway;
adding ID identification including a log identity ID and a log link ID in a distributed log file, namely recording and identifying environmental information of a log generation node when a gateway intercepts the log file, so that the generated log file has the log identity ID, the log link ID and the like of the generation environment of the log file, and storing the log file with the ID in a server of a log query platform, wherein the log file can be a log memory queue in the server; at this time, the total log file is still collected in the log memory queue.
When a query instruction is obtained, firstly analyzing the query instruction to obtain user information of the query instruction, obtaining a log Identity (ID) matched with the user information according to the user information obtained by analysis, copying a log file in a log memory queue according to the log Identity (ID), and independently storing the log file in another user message queue;
and acquiring a result log file from the user message queue according to the query condition of the query instruction.
The scheme does not limit the monitoring and analysis of the user on the whole data, and generally needs to monitor and analyze for a period of time after the query instruction when the log is queried actually, so that the advantage of saving the data volume can be fully embodied.
Example four
The third difference from the third embodiment is that the embodiment separately sets a link of the method for acquiring the monitoring instruction, and a time node for specifically acquiring the monitoring instruction is not limited to a time node for issuing the query instruction, and may be any time node before the query. For example, if the user is a paid enterprise user and has an enterprise user ID, the monitoring instruction may be started after a time node specified by the user, the log file associated with the user and having the enterprise user ID may be screened, and the log file obtained by monitoring may be stored in the user message queue. The user can monitor the log file and log change information in the user message queue.
The embodiment provides a data processing method of a distributed log, as shown in fig. 2, which is applied to a distributed log system, and includes the following steps:
s201: intercepting and generating a distributed log file at a gateway;
s202: adding an ID (identity) in a distributed log file, wherein the ID comprises a user Identity (ID), a machine ID, a link ID, a service ID and an analysis software link ID, and storing the log file with the ID in a log memory queue in a server of a log query platform;
s203: acquiring a monitoring instruction, analyzing user information of the monitoring instruction, acquiring an ID (identity) matched with the user information, screening a distributed log file according to the ID and acquiring a user log file related to the user information;
s204: sending the user log file to a user server corresponding to the monitoring instruction;
s205: acquiring a query instruction, and acquiring a user result log file from a user server according to a query condition of the query instruction;
s206: and displaying the user result log file on the terminal of the user.
By the method, log monitoring of the log files related to the user can be provided for the specific user in real time, when the user inquires the log, the log files are screened according to the IDs added to the gateway, so that the processed log files are the logs required by the user or the related logs, and when the user inquires the log, the target is not the full log file but the specific log file related to the user, so that the number of the log files in a user message queue is greatly reduced, the monitoring purpose is improved, the processing amount of the log files is reduced, and the monitoring efficiency and the query efficiency are improved.
Furthermore, in the scheme, the log file related to the user can be sent to the user terminal and stored in the user terminal, so that the data pressure of the server of the log query platform is reduced.
EXAMPLE five
The embodiment provides a data processing method of a distributed log, which comprises the following steps:
acquiring a distributed log file, adding an ID (identity) to the distributed log file and acquiring the log file with the ID;
the log files with the IDs are sorted according to the ID identifications and then stored in a server of a log query platform;
acquiring a query instruction, and acquiring a result log file from a server according to a query condition of the query instruction;
and displaying the result log file on a Web terminal of the log query platform.
The embodiment provides a data processing method of a distributed log, which is applied to a distributed log system and comprises the following steps:
intercepting and generating a distributed log file at a gateway; adding an ID (identity) in a distributed log file, wherein the ID comprises a user Identity (ID), a machine ID, a link ID, a service ID and an analysis software link ID, and storing the log file with the ID in a log memory queue in a server of a log query platform;
acquiring a monitoring instruction, analyzing user information of the monitoring instruction, acquiring an ID (identity) matched with the user information, screening a distributed log file according to the ID and acquiring a user log file related to the user information; the step of acquiring the ID matched with the user information comprises the following steps: obtaining the category of ID identification, wherein the category of ID identification comprises: tenant ID, enterprise ID, operational user ID; analyzing the user information, and acquiring the tenant identity, the enterprise identity and the operation user identity of the user information; and obtaining the ID identification matched with the user information according to the type of the ID identification and the analysis result of the user information.
Sending the user log file to a user server corresponding to the monitoring instruction;
acquiring a query instruction, and acquiring a user result log file from a user server according to a query condition of the query instruction;
the user result log file is displayed on the user's terminal.
If the user result log level is the debugging DEBUG level, the font of the classification log file is marked as gray #909399 by using a color style;
if the user result log level is the warning WARN level, the font of the classification log file is marked as yellow # E6A23C by using a color style;
if the user result log level is the ERROR ERROR level, the font of the classification log file is marked as red # F56C6C by using a color style;
if the user result log level is a normal level, the font of the classification log file is black;
a log display using a < pre > tag tape style is displayed in a text box of a display screen of the user terminal.
Acquiring the color of the classification log file;
acquiring the color scrolling speed of the classification log file according to the color of the classification log file;
displaying the classification log file in a scrolling text box according to the color scrolling speed;
the scrolling speed of the red log file is greater than that of the yellow log file, the scrolling speed of the yellow log file is greater than that of the gray log file, and the scrolling speed of the gray log file is greater than that of the black log file.
According to the scheme, the scrolling speed of the scrolling text box is acquired by identifying the color of the log file, when the log files of warning types and error types are more, the monitored log file data can be displayed in the colors of red, yellow and the like, and the log file data can be displayed at a lower scrolling speed through color identification, so that a user can notice and read the log content more easily. Meanwhile, the scrolling speed is confirmed only through color identification without analyzing the characteristics of the log, and the setting of the scrolling speed is faster and more convenient.
Furthermore, the log analysis result of the user result log file can be obtained according to the color category of the classification log file by obtaining the color category of the classification log file.
Besides obtaining the scrolling speed by using the color, the analysis result of the source problem traced back by the log file, such as the importance level of the source problem, the distribution state of the source problem, etc., can be further obtained by the color proportion in a time period or the color proportion of the display content of a display screen.
EXAMPLE six
In the step of adding the ID identification, the link ID of the analysis software can be added, and the distributed log files are screened according to the ID identification and the user log files related to the user information are obtained;
acquiring the operation/processing occupation time of a user log file by analyzing the software link ID;
acquiring the indentation width of the log file according to the operation/processing occupation time of the user log file, wherein the longer the operation/processing occupation time of the user log file is, the longer or the shorter the indentation width of the corresponding log file is;
and acquiring a performance analysis result of the user log file according to the indentation width corresponding to the user log file.
The log file is analyzed and processed by using the display result on the display screen of the Web terminal or the user terminal, the log file is analyzed not based on the data of the log file but based on the display format of the display screen, the data volume is smaller, the data processing pressure is smaller, the result is more visual, and the performance analysis result of the user log file is obtained while the user can conveniently watch the log file. For example, the operation duration of the user log file, the processing duration of the user log file, the operation duration ratio of the user log file, the screening of the oversized log file whose operation duration exceeds the threshold value, and the like.
EXAMPLE seven
Existing log file systems are generally classified based on programs, and after a monitoring task or an inquiry task is triggered, the classified log files are analyzed based on task conditions. The processing method cannot perform automatic context search based on tasks, and cannot perform clustering processing on associated log files, so that data integrity is insufficient.
The embodiment provides a data processing method of a distributed log, which comprises the following steps:
acquiring a distributed log file, adding an ID (identity) to the distributed log file and acquiring the log file with the ID;
the log files with the IDs are sorted according to the ID identifications and then stored in a server of a log query platform;
acquiring a monitoring instruction, analyzing user information of the monitoring instruction, and acquiring an ID (identity) identifier matched with the user information, wherein the ID identifier comprises a link ID;
if the distributed log file has no link ID, giving the link ID to the log file;
screening the distributed log files according to the user identity IDs, clustering the screening results according to the link IDs and obtaining clustering result log files, wherein the clustering comprises same-thread clustering, same-machine clustering, same-service clustering and same-environment clustering;
sending the clustering result log file to a user server corresponding to the monitoring instruction;
acquiring a query instruction, acquiring a query condition and a thread ID corresponding to the query instruction, acquiring a collinear Cheng Julei file or a same-machine cluster file or a same-service cluster file or a same-environment cluster file in a cluster result file according to the thread ID, performing file search on a collinear Cheng Julei file or a same-machine cluster file or a same-service cluster file or a same-environment cluster file according to the query condition, and acquiring a user result log file;
the user result log file is displayed on the user's terminal.
And the clustering processing based on the link ID can realize the clustering processing of log files in the same environment, the same machine and the same thread, and when in query, the context is searched based on the link ID and the thread ID, all log files related to the thread are found and the monitoring query processing is carried out.
Example eight
Acquiring the text box length of a text box of a display screen of a user terminal in the display screen of the user terminal or a Web terminal; acquiring the text length of the classification log file;
hiding the text of the classification log file exceeding the length of the text box according to the length of the text box;
establishing a hidden connection button for hiding the classification log file; and if a button click instruction is acquired, displaying all text contents of the hidden classification log file through a popup window.
The display effect of the display screen is simplified by hiding the length of the super-large text, and the display efficiency is improved.
Example nine
The technical problem to be solved by the scheme is that the existing distributed log system built through ELK open source software stores log files generated by each node after being collected through a message queue, due to the characteristics of the distributed log system, the data volume of the generated distributed log is very large, the monitoring data generated by monitoring the distributed log is also very large, and due to distributed deployment, the source tracing and positioning of the finally generated log files are relatively disordered and complex.
In order to solve the above technical problem, the present embodiment provides a data processing apparatus for distributed logs, as shown in fig. 3, including: the ID adding module 100 is used for obtaining a distributed log file, adding an ID into the distributed log file and obtaining a log file with the ID;
the sequencing storage module 200 is used for storing the log files with the IDs after sequencing according to the ID identifications in a server of the log query platform;
the query module 300 is configured to obtain a query instruction, and obtain a result log file from a server according to a query condition of the query instruction;
and a display module 400, configured to display the result log file on a Web terminal of the log query platform.
Example ten
FIG. 4 is a diagram illustrating an internal structure of a computer device in one embodiment. The computer device may specifically be a terminal, and may also be a server. As shown in fig. 4, the computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may further store a computer program that, when executed by the processor, causes the processor to implement a data processing method of the distributed log. The internal memory may also have stored therein a computer program that, when executed by the processor, causes the processor to perform a data processing method of the distributed log. It will be appreciated by those skilled in the art that the configuration shown in fig. 4 is a block diagram of only a portion of the configuration associated with the present application, and is not intended to limit the computing device to which the present application may be applied, and that a particular computing device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In an embodiment, a computer device is proposed, comprising a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to perform the steps of:
acquiring a distributed log file, adding an ID (identity) to the distributed log file and acquiring the log file with the ID;
the log files with the IDs are sorted according to the ID identifications and then stored in a server of a log query platform;
acquiring a query instruction, and acquiring a result log file from a server according to a query condition of the query instruction;
and displaying the result log file on a Web terminal of the log query platform.
In one embodiment, a computer-readable storage medium is proposed, in which a computer program is stored which, when executed by a processor, causes the processor to carry out the steps of:
acquiring a distributed log file, adding an ID (identity) to the distributed log file and acquiring the log file with the ID;
the log files with the IDs are sorted according to the ID identifications and then stored in a server of a log query platform;
acquiring a query instruction, and acquiring a result log file from a server according to a query condition of the query instruction;
and displaying the result log file on a Web terminal of the log query platform.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a non-volatile computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the program is executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent application shall be subject to the appended claims.
Claims (8)
1. A data processing method of a distributed log is applied to a distributed log system and is characterized by comprising the following steps:
intercepting and generating a distributed log file at a gateway, and adding an ID (identity) in the distributed log file, wherein the ID comprises a time ID, a log identity ID and a log link ID;
sorting the log files with the IDs according to the time IDs and storing the log files in a memory message queue in a server of a log query platform;
acquiring a query instruction, analyzing user information of the query instruction, acquiring a log Identity (ID) matched with the user information, screening a distributed log file according to the acquired log Identity (ID) and storing the distributed log file in a user message queue of a server;
acquiring a result log file from a user message queue of the server according to the query instruction;
monitoring a log message queue in a server of a log query platform according to a query condition of a query instruction;
defining a listening class MQLogListener of the log message queue, the listening class MQLogListener being used for listening log changes of the log message queue, two long arrays having a first array length being defined in the MQLogListener: the first long array and the second long array are used for circularly storing a result log file;
storing the result log file in a first long array or a second long array and detecting whether the length of the stored long array exceeds the length of the first array;
if the length of the first array is not exceeded, monitoring a log message queue in a text box of a display screen of the Web terminal by polling and monitoring an inquiry interface TailQuery in a timing manner, wherein the polling conditions of the inquiry interface TailQuery in the timing manner comprise a page filtering condition, a final log ID and a maximum log length;
when a timing polling monitoring query interface TailQuery receives a query instruction, according to the last log ID of a Web terminal display screen, adopting the starting point position of a binary bit length array to start querying a result log file from the starting point position of a long array, and simultaneously filtering the result log file by using keywords until the long array reaches the maximum length;
performing splicing operation on the monitored and inquired logs by using an array push method to obtain a first spliced log file;
screening the first spliced log file through a fixed-length array, defining the fixed-length array through a Web terminal, and deleting the logs exceeding the fixed-length array if the first spliced log file exceeds the size of the fixed-length array;
splicing the filtered first spliced log files by using an array join method to obtain a complete log body, and taking the complete log body as a result log file;
acquiring the grade of the result log, and marking the fonts of the classified log files into different colors by using a color style according to the grade of the result log;
displaying a log in a text box of a Web terminal display screen by using a < pre > tag tape style;
if the time sequence query is selected on the page of the scroll text box of the Web terminal display screen, the log display is automatically set to the top for display by the setTimeout delay of 50 milliseconds and by using a scrollTop (0), and the classified log files subjected to color rendering can be displayed in sequence from far to near;
if the page of the scroll text box of the display screen of the Web terminal selects time reverse query, the log display is automatically scrolled to the lowest by using scrollTop (1000000) after the setTimeout is delayed for 50 milliseconds, and the classified log file subjected to color rendering can be displayed from near to far.
2. The data processing method of the distributed log according to claim 1, wherein: intercepting and generating a distributed log file at a gateway, adding an ID (identity) identifier into the distributed log file, wherein the ID identifier comprises a time ID, a log identity ID and a log link ID, sorting the log file with the ID according to the time ID, storing the sorted log file in a memory message queue in a server of a log query platform, acquiring a query instruction, analyzing user information of the query instruction, acquiring a log identity ID matched with the user information, screening the distributed log file according to the acquired log identity ID, storing the distributed log file in a user message queue of the server, and acquiring a result log file from the user message queue of the server according to the query instruction, wherein the steps comprise:
intercepting and generating a distributed log file at a gateway;
adding ID identification in a distributed log file, wherein the ID identification comprises a time ID, a user identity ID, a machine ID, a link ID, a service ID and an analysis software link ID;
sorting the log files with the ID identifications according to the time IDs and storing the log files in a memory message queue in a server of a log query platform;
acquiring a monitoring instruction, analyzing user information of the monitoring instruction, acquiring an ID (identity) matched with the user information, screening a distributed log file according to the ID, acquiring a user log file related to the user information, and storing the user log file in a first user message queue of a server or a second user message queue of the user server;
acquiring a query instruction, and acquiring a user result log file from a server or a user result log file from a user server according to a query condition of the query instruction;
the user result log file is displayed on the user's terminal.
3. The data processing method of the distributed log according to claim 2, wherein: the step of acquiring the ID matched with the user information comprises the following steps:
obtaining the category of ID identification, wherein the category of ID identification comprises: tenant ID, enterprise ID, operational user ID;
analyzing the user information, and acquiring the tenant identity, the enterprise identity and the operation user identity of the user information;
and obtaining the ID identification matched with the user information according to the type of the ID identification and the analysis result of the user information.
4. The data processing method of the distributed log according to claim 3, wherein:
if the user result log level is the debugging DEBUG level, marking the font of the classification log file as a first color by using a color style;
if the user result log level is the warning WARN level, the font of the classification log file is marked as a second color by using a color style;
if the user result log level is the ERROR ERROR level, the font of the classification log file is marked to be a third color by using a color style;
if the user result log level is a normal level, the font of the classification log file is a fourth color;
a log display using a < pre > tag tape style is displayed in a text box of a display screen of the user terminal.
5. The data processing method of the distributed log according to claim 4, wherein: acquiring the color of the classification log file;
acquiring the color scrolling speed of the classification log file according to the color of the classification log file;
displaying the classification log file in a scrolling text box according to the color scrolling speed;
the scrolling speed of the log file of the first color is greater than that of the log file of the second color, the scrolling speed of the log file of the second color is greater than that of the log file of the third color, and the scrolling speed of the log file of the third color is greater than that of the log file of the fourth color;
and obtaining color type information of the user result log file, wherein the color type information comprises the proportion of the first color, the proportion of the second color, the proportion of the third color and the proportion of the fourth color, and obtaining a log analysis result according to the color type information of the user result log file.
6. A data processing apparatus for distributed logging, comprising:
the ID adding module is used for intercepting and generating a distributed log file at a gateway, and adding ID identification in the distributed log file, wherein the ID identification comprises a time ID, a log identity ID and a log link ID;
the sequencing storage module is used for sequencing the log files with the IDs according to the time IDs and then storing the log files in a memory message queue in a server of the log query platform;
the query module is used for acquiring a query instruction, analyzing the user information of the query instruction, acquiring a log Identity (ID) matched with the user information, screening a distributed log file according to the acquired log Identity (ID) and storing the distributed log file in a user message queue of the server;
the display module is used for acquiring a result log file from a user message queue of the server according to the query instruction;
monitoring a log message queue in a server of a log query platform according to a query condition of a query instruction;
defining a listening class MQLogListener of the log message queue, the listening class MQLogListener for listening for log changes to the log message queue, defining two long arrays having a first array length in MQLogListener: the first long array and the second long array are used for circularly storing a result log file;
storing the result log file in a first long array or a second long array and detecting whether the length of the stored long array exceeds the length of the first array;
if the length of the first array is not exceeded, monitoring a log message queue in a text box of a display screen of the Web terminal by polling and monitoring an inquiry interface TailQuery in a timing manner, wherein the polling conditions of the inquiry interface TailQuery in the timing manner comprise a page filtering condition, a final log ID and a maximum log length;
when a timing polling monitoring query interface TailQuery receives a query instruction, according to the last log ID of a Web terminal display screen, adopting the starting point position of a binary bit length array to start querying a result log file from the starting point position of a long array, and simultaneously filtering the result log file by using keywords until the long array reaches the maximum length;
performing splicing operation on the monitored and inquired logs by using an array push method to obtain a first spliced log file;
screening the first spliced log file through a fixed-length array, defining the fixed-length array through a Web terminal, and deleting the logs exceeding the fixed-length array if the first spliced log file exceeds the size of the fixed-length array;
splicing the filtered first spliced log files by using an array join method to obtain a complete log body, and taking the complete log body as a result log file;
acquiring the grade of the result log, and marking the fonts of the classified log files into different colors by using a color style according to the grade of the result log;
displaying a log in a text box of a Web terminal display screen by using a < pre > tag tape style;
if the time sequence query is selected on the page of the scroll text box of the Web terminal display screen, the log display is automatically set to the top for display by the setTimeout delay of 50 milliseconds and by using a scrollTop (0), and the classified log files subjected to color rendering can be displayed in sequence from far to near;
if the time for searching is selected in the reverse order from the page of the scrolling text box of the display screen of the Web terminal, the log display is automatically scrolled to the lowest by using scrollTop (1000000) after the setTimeout is delayed for 50 milliseconds, and the classified log file subjected to color rendering can be displayed from near to far.
7. A computer-readable storage medium, storing a computer program which, when executed by a processor, causes the processor to carry out the steps of the method according to any one of claims 1 to 5.
8. A computer device comprising a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211269620.6A CN115357689B (en) | 2022-10-18 | 2022-10-18 | Data processing method, device and medium of distributed log and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211269620.6A CN115357689B (en) | 2022-10-18 | 2022-10-18 | Data processing method, device and medium of distributed log and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115357689A CN115357689A (en) | 2022-11-18 |
| CN115357689B true CN115357689B (en) | 2023-02-17 |
Family
ID=84008168
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211269620.6A Active CN115357689B (en) | 2022-10-18 | 2022-10-18 | Data processing method, device and medium of distributed log and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115357689B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102519006B1 (en) * | 2022-11-21 | 2023-04-07 | 주식회사 에스티씨랩 | Log data management method and server for improving search speed of log data based on digital |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109815201A (en) * | 2018-12-15 | 2019-05-28 | 深圳壹账通智能科技有限公司 | Log processing method and relevant device based on message queue and search engine |
| CN110231998A (en) * | 2019-06-13 | 2019-09-13 | 泰康保险集团股份有限公司 | Detection method, device and the storage medium of distributed timing task |
| CN113449339A (en) * | 2021-06-29 | 2021-09-28 | 平安资产管理有限责任公司 | Log collection method, system, computer device and computer readable storage medium |
| CN114138604A (en) * | 2021-12-01 | 2022-03-04 | 浪潮云信息技术股份公司 | Transaction log processing method and system for distributed database |
| CN114385551A (en) * | 2021-12-20 | 2022-04-22 | 武汉物易云通网络科技有限公司 | Log time-sharing management method, device, equipment and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8122050B2 (en) * | 2008-04-16 | 2012-02-21 | International Business Machines Corporation | Query processing visualization system and method of visualizing query processing |
-
2022
- 2022-10-18 CN CN202211269620.6A patent/CN115357689B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109815201A (en) * | 2018-12-15 | 2019-05-28 | 深圳壹账通智能科技有限公司 | Log processing method and relevant device based on message queue and search engine |
| CN110231998A (en) * | 2019-06-13 | 2019-09-13 | 泰康保险集团股份有限公司 | Detection method, device and the storage medium of distributed timing task |
| CN113449339A (en) * | 2021-06-29 | 2021-09-28 | 平安资产管理有限责任公司 | Log collection method, system, computer device and computer readable storage medium |
| CN114138604A (en) * | 2021-12-01 | 2022-03-04 | 浪潮云信息技术股份公司 | Transaction log processing method and system for distributed database |
| CN114385551A (en) * | 2021-12-20 | 2022-04-22 | 武汉物易云通网络科技有限公司 | Log time-sharing management method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115357689A (en) | 2022-11-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9025890B2 (en) | Information classification device, information classification method, and information classification program | |
| US11042525B2 (en) | Extracting and labeling custom information from log messages | |
| US8886660B2 (en) | Method and apparatus for tracking a change in a collection of web documents | |
| CN111046645A (en) | Method and device for generating article, computer equipment and storage medium | |
| CN114417197A (en) | Access record processing method and device and storage medium | |
| CN107870849B (en) | Processing method and device of test log | |
| CN115357689B (en) | Data processing method, device and medium of distributed log and computer equipment | |
| CN113254255B (en) | Cloud platform log analysis method, system, device and medium | |
| CN111400361A (en) | Data real-time storage method and device, computer equipment and storage medium | |
| CN113688288A (en) | Data association analysis method and device, computer equipment and storage medium | |
| CN106294406B (en) | Method and equipment for processing application access data | |
| US8140671B2 (en) | Apparatus and method for sampling security events based on contents of the security events | |
| CN114968959A (en) | Log processing method, log processing device and storage medium | |
| CN111966339B (en) | Buried point parameter input method and device, computer equipment and storage medium | |
| CN111459796B (en) | Automated testing method, apparatus, computer device and storage medium | |
| CN110362478B (en) | Application upgrade test method and device, computer equipment and storage medium | |
| CN110716866A (en) | Code quality scanning method and device, computer equipment and storage medium | |
| CN114116811B (en) | Log processing method, device, equipment and storage medium | |
| CN113360313B (en) | Behavior analysis method based on massive system logs | |
| CN114861074A (en) | User data analysis method and system | |
| CN114218569A (en) | Data analysis method, device, equipment, medium and product | |
| CN113255742A (en) | Policy matching degree calculation method and system, computer equipment and storage medium | |
| CN115129927B (en) | Monitoring video stream backtracking method, electronic equipment and storage medium | |
| CN114840672A (en) | Log analysis method, device, equipment and storage medium | |
| CN114328076B (en) | Log information extraction method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |