WO2019134341A1

WO2019134341A1 - Log text processing method and apparatus, and storage medium

Info

Publication number: WO2019134341A1
Application number: PCT/CN2018/089870
Authority: WO
Inventors: 乐伟
Original assignee: 平安科技（深圳）有限公司
Priority date: 2018-01-05
Filing date: 2018-06-05
Publication date: 2019-07-11
Also published as: CN108427626A

Abstract

Provided are a log text processing method and apparatus, and a computer-readable storage medium. The method comprises the following steps: acquiring a log file path, opening a log file, and locating a current log text by using a cursor; tracking a log text increment regularly using the cursor; extracting log text paragraphs, that satisfy a query parameter, row by row using the cursor; and compiling statistics on the extracted log text paragraphs and outputting a statistical result of a pre-determined statistical item. By means of the present application, a log text increment can be tracked, a log text satisfying a query parameter can be quickly extracted, and the statistics of statistical items of a log can be compiled in a timely manner.

Description

Log text processing method, device and storage medium

Priority claim

This application claims priority to Chinese Patent Application No. 201810011139.4, filed on Jan. 5, 2018, entitled "Log Text Processing Method, Apparatus, and Storage Medium", the contents of which are incorporated herein by reference. In the application.

Technical field

The present application relates to the field of computer technologies, and in particular, to a log text processing method, apparatus, and storage medium.

Background technique

The log can record various information about the running system or software, including request time, Internet Protocol (IP) address, request type, number of data transfer bytes, and status code. In actual work, the most critical part of the log is often the newly generated several lines or hundreds of lines of text. The latest generated log text can be used to judge the running status of the system or software, which tracks, queries and extracts the log text increments. A higher requirement was put forward.

Flume is the most commonly used log collection technology in the prior art. It generally uses Spooling Directory Source or Exec Source to extract logs. However, Spooling Directory Source does not support appending the extracted log text. When new log content is generated, only The new log content can be extracted by creating a new log file. If this method is used to extract the log text increment, a large number of small files will be generated in the log directory; and the Exec Source extracts the log text increment reliably. The sex is low, and it is prone to the failure of extraction.

Summary of the invention

To solve the deficiencies of the prior art, the present application provides a log text processing method and apparatus, and a computer readable storage medium, which can track and extract log texts by using a cursor to accurately count various statistical items of the log.

To achieve the above objective, the present application provides a log text processing method, which is applied to an electronic device, and the method includes:

Text positioning step: obtain the log file path, open the log file, point the cursor to the last line of the current log text, and record the number of lines of the current log text as the cursor reference value;

Incremental tracking step: set the timing execution value, advance the cursor to the last line of the log text increment, and update the cursor reference value according to the number of rows of the log text increment;

Query extraction step: setting the query parameters, and using the cursor to extract the log text paragraphs satisfying the query parameters from the log text;

Statistical step: Statistics are performed on the extracted log text segments, and the statistical results of the predetermined statistical items are output.

To achieve the above object, the present application further provides an electronic device including a memory and a processor, wherein the memory includes a log text processing program, and when the log text processing program is executed by the processor, the following steps are implemented:

In addition, in order to achieve the above object, the present application further provides a computer readable storage medium, which includes a log text processing program, and when the log text processing program is executed by a processor, the following steps are implemented:

The log text processing method, the electronic device and the computer readable storage medium proposed by the application use a cursor to locate the current log text, track the log text increment, and extract the log text satisfying the query parameter, according to the extracted log text. Statistics are collected for each statistical item of the log. Since the timing execution value of the tracking log text increment can be freely set, the use of the cursor does not affect the content of the log text, so the application can track the log text without intrusion, accurately extract the query result, and realize flexible log text. And handle it efficiently.

DRAWINGS

1 is a schematic diagram of an operating environment of a preferred embodiment of an electronic device of the present application;

2 is a program block diagram of the log text processing program of FIG. 1;

3 is a flowchart of a first preferred embodiment of a log text processing method of the present application;

4 is a flow chart of a second preferred embodiment of a log text processing method of the present application.

The implementation, functional features and advantages of the present application will be further described with reference to the accompanying drawings.

Detailed ways

Those skilled in the art will appreciate that embodiments of the present application can be implemented as a method, apparatus, device, system, or computer program product. Accordingly, the application can be embodied in a complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

The principles and spirit of the present application are described below with reference to a number of specific embodiments. It is understood that the specific embodiments described herein are merely illustrative of the application and are not intended to be limiting.

1 is a schematic diagram of an operating environment of a preferred embodiment of an electronic device of the present application.

The electronic device 1 may be a terminal device having a storage and computing function such as a server, a smart phone, a tablet computer, a portable computer, or a desktop computer.

The electronic device 1 includes a memory 11, a processor 12, a network interface 13, and a communication bus 14. The network interface 13 can optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The communication bus 14 is used to implement connection communication between the above components.

The memory 11 includes at least one type of readable storage medium. The at least one type of readable storage medium may be a non-volatile storage medium such as a flash memory, a hard disk, a multimedia card, a card type memory, or the like. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1, such as a hard disk of the electronic device 1. In other embodiments, the readable storage medium may also be an external memory 11 of the electronic device 1, such as a plug-in hard disk equipped on the electronic device 1, a smart memory card (SMC). , Secure Digital (SD) card, Flash Card, etc.

In the present embodiment, the readable storage medium of the memory 11 is used to store the log text processing program 10, the log file, and other data to which the processor 12 executes the program code of the log text processing program 10 and the last output data. Wait.

The processor 12, in some embodiments, may be a Central Processing Unit (CPU), microprocessor or other data processing chip for running program code or processing data stored in the memory 11, such as performing log text processing. Program 10 and so on.

1 shows only the electronic device 1 having the components 11-14 and the log text processing program 10, but it should be understood that not all illustrated components may be implemented, and more or fewer components may be implemented instead.

Optionally, the electronic device 1 may further include a user interface, and the user interface may include an input unit such as a keyboard, a voice input device such as a microphone, a device with a voice recognition function, a voice output device such as an audio, a headphone, and the like. . Optionally, the user interface may also include a standard wired interface and a wireless interface.

Optionally, the electronic device 1 may further include a display, which may also be referred to as a display screen or a display unit. In some embodiments, it may be an LED display, a liquid crystal display, a touch liquid crystal display, and an Organic Light-Emitting Diode (OLED) display. The display is used to display information processed in the electronic device 1 and a user interface for displaying visualizations.

Optionally, the electronic device 1 further comprises a touch sensor. The area provided by the touch sensor for the user to perform a touch operation is referred to as a touch area. Further, the touch sensor described herein may be a resistive touch sensor, a capacitive touch sensor, or the like. Moreover, the touch sensor includes not only a contact type touch sensor but also a proximity type touch sensor or the like. Furthermore, the touch sensor may be a single sensor or a plurality of sensors arranged, for example, in an array. The user can start the log text processing program 10 by touching the touch area.

In addition, the area of the display of the electronic device 1 may be the same as or different from the area of the touch sensor. Optionally, a display is stacked with the touch sensor to form a touch display screen. The device detects a user-triggered touch operation based on a touch screen display.

The electronic device 1 may further include a radio frequency (RF) circuit, a sensor, an audio circuit, and the like, and details are not described herein.

In the operating environment diagram of the preferred embodiment of the electronic device 1 shown in FIG. 1, the memory 11 including the readable storage medium may include an operating system, a log text processing program 10, a log file, and the like. When the processor 12 executes the log text processing program 10 in the memory 11, the following steps are implemented:

For details, please refer to the following FIG. 2 for a description of the program block diagram of the log text processing program 10 and FIG. 3 and FIG. 4 for a flow chart of the preferred embodiment of the log text processing method.

Referring to FIG. 2, it is a program block diagram of the log text processing program 10 of FIG. In the present embodiment, the log text processing program 10 is divided into a plurality of modules, which are stored in the memory 11 and executed by the processor 12 to complete the present application. A module as referred to in this application refers to a series of computer program instructions that are capable of performing a particular function.

The log text processing program 10 can be divided into: a positioning module 110, a tracking module 120, a query extraction module 130, and a statistics module 140.

The positioning module 110 is configured to locate the current log text. The storage path of the log file can be variously selected. Different package managers (Package Manager) will put the log files in different locations, but the configuration files can be used to obtain the storage path of the log files, thereby opening the log file. Many programs do not have the ability to manipulate a collection, and the amount of data in the log is often large, so you need to use a cursor to operate line by line. After opening the log file, the positioning module 110 uses the OPEN statement to open the cursor, so that the cursor points to the first line of the current log text, and then uses the FETCH statement to advance the cursor to the last line of the current log text, and records the number of lines of the current log text as a cursor. The benchmark value, and finally use the CLOSE statement to close the cursor. The minimum value of the cursor reference value, that is, the initial value of the cursor reference value, is recorded as 0. It can be understood that the cursor is equivalent to a pointer and has the ability to manipulate multi-line log text, and can process multiple lines of log text one line at a time or multiple lines at a time. In another embodiment, the cursor can also be controlled by a cursor FOR loop instead of the OPEN, FETCH, and CLOSE statements.

The tracking module 120 is configured to periodically track the log text increments to obtain the number of rows of the log text increment. The log can be updated in real time, and the log text increment may be generated at any time. After the current log text is located, the timing execution value is set, and the tracking module 120 periodically advances the cursor to the last line of the log text increment, and according to the log text increment. The number of rows updates the cursor reference value. The timing execution value is in seconds. For example, if the timing execution value is 60 seconds, the tracking module 120 advances the cursor to the last line of the log text increment every 60 seconds, and updates according to the number of rows of the log text increment. Cursor reference value. It can be understood that the amount of change of the cursor reference value is equal to the number of rows of the log text increment.

The query extraction module 130 is configured to extract log text that satisfies the query parameter by row. Since the log can record operation information such as the request time, and feedback information such as a status code can also be recorded, the query parameter can include a time period and a keyword. When the query parameter is a time segment, the query extraction module 130 controls the cursor to extract the log text generated in the time segment row by row; when the query parameter is a keyword, the query extraction module 130 controls the cursor, and the keyword is included. The log text is extracted line by line; when the query parameter is combined with the time segment and the keyword, the query extraction module 130 controls the cursor, and extracts the log text containing the keyword in the log text generated in the time segment row by line.

It can be understood that the cursor is used to locate the current log text, and the log text increment is tracked. When the log text segment that satisfies the query parameter is extracted, the cursor is used as an external tool, and no log is generated. The effect does not change the content of the log text.

The statistics module 140 is configured to perform statistics on the extracted log text segments, and output statistical results of the predetermined statistical items. The format of the log can be customized by the user. The user can obtain the record information that is obtained through the customized log format. From the record information, the statistics of the log items can be counted. The record information in the log may include a requester IP address, a data transmission byte number, a status code, a network protocol name and a version, etc., and the predetermined statistical item may include a total amount of data transmission and a number of rows of log text increment, and may also Including Unique Visitor (UV), number of request successes, number of request failures, reason for request failure, number of times data transmission was interrupted, and so on. For example, the number of increments of the log text can be counted by the amount of change of the cursor reference value, and the total amount of data transmission in a certain period of time can be counted according to the number of bytes of data transmission. The UV can be counted according to the IP address of the requester, and the status code can be used. The number of request successes, the number of request failures, and the reason for the request failure. It can be understood that the data transmission here includes data uploading and data downloading.

Specifically, in this embodiment, the log is a website access log, and each line of text of the website access log records a status code, and the status code is also called an error code, which is allocated for each request received by the website server. A three-digit code that indicates whether the request was successful and the type of error encountered. When the status code is 200, it means that the website server successfully responds to the browser's request, everything is normal; when the status code is 400, it means that the request initiated by the browser is wrong, for example, the request contains a syntax error, etc.; when the status code is 404 When it is displayed, the web server cannot find the web page requested by the browser. If the query extraction module 130 extracts the log text increments within a certain period of time, the statistics module 140 may use the status code to count the number of times the web server processes the browser request successfully during the time period, the number of failures, and the web server processing the browser. If the query extraction module 130 uses the keyword as the query parameter, for example, the query parameter is 200, and all the logs with the status code of 200 are extracted, the statistics module 140 can use the status code to count the successful response of the website server to the browser. The total number of requests. The above explanation about the use of the status code to count the number of request successes, the number of request failures, and the reason for the failure of the request is only a partial example, and is not exhaustive. The website operation and maintenance personnel can judge whether the running status of the website server is normal according to the statistical result output by the statistics module 140.

In this embodiment, each line of text of the website access log also records the number of bytes sent by the website server to the client, that is, the amount of data downloaded by the client from the website server, and the statistics module 140 adds the values to count the server. The total amount of data sent by the server during a certain period of time, such as one day, one week, or one month. In addition, by comparing the number of bytes sent by the website server to the client and the size of the corresponding file, the statistics module 140 can determine whether the data transmission process is interrupted, and count the number of times the data transmission is interrupted within a certain period of time. . Similarly, the statistics module 140 may use the information recorded by the log to output statistical results of other predetermined statistical items for reference by developers, operation and maintenance personnel, and the like.

Referring to FIG. 3, it is a flowchart of a first preferred embodiment of a log text processing method of the present application. The following steps of implementing the log text processing method when the processor 12 of the electronic device 1 executes the log text processing program 10 stored in the memory 11:

Step S10: Obtain a log file path, open a log file, and use the positioning module 110 to point the cursor to the last line of the current log text to locate the current log text. The storage path of the log file can be obtained through the configuration file. After the log file is opened, the positioning module 110 opens the cursor through the OPEN statement, points the cursor to the first line of the current log text, and then advances the cursor to the end of the current log text through the FETCH statement. One line, the number of lines of the current log text is recorded as the cursor reference value, and the positioning of the current log text is completed. In another embodiment, the cursor can be controlled by a cursor FOR loop instead of the OPEN statement and the FETCH statement.

In step S20, the timing execution value is set, and the cursor is advanced to the last line of the log text increment by the tracking module 120, and the log text increment is tracked periodically. The timing execution value is in seconds, and if the timing execution value is 60 seconds, the tracking module 120 advances the cursor to the last row of the log text increment every 60 seconds, and updates according to the number of rows of the log text increment. The cursor reference value, that is, the amount of change in the cursor reference value is equal to the number of rows in the log text increment, enabling periodic tracking of log text increments. In one embodiment, the value representing the number of rows of log text increments is always displayed on the display and periodically updated with a timing execution value.

Step S30, setting a query parameter, and using the query extraction module 130 to extract a log text paragraph that satisfies the query parameter. The query parameter may be a time period or a keyword. Therefore, the query extraction module 130 may be used to extract a log text added to a certain period of time, or may extract a log text line containing certain keywords.

For convenience of description, we refer to the log text satisfying the query parameters as a result set. According to the query parameters set by the user, the query extraction module 130 controls the cursor to point to the first row of the result set, and extracts the log text of the row from the result set. , then advance the cursor to the next row of the result set until all log text lines that satisfy the query parameters are extracted, the entire result set.

In step S40, the statistical analysis module 140 performs statistical analysis on the extracted log text paragraphs, and outputs statistical results of the predetermined statistical items. The predetermined statistical items include the total amount of data transmission, the number of rows of the log text increment, the number of independent visitors, the number of successful requests, the number of failed requests, the reason for the failure of the request, the number of times the data transmission is interrupted, and the like. For the specific method and function of the statistical analysis, please refer to the above detailed description of the statistical analysis module 140.

The log text processing method proposed in this embodiment uses a cursor to locate the current log text, tracks the log text increment, accurately extracts the log text that satisfies the query parameter, and performs statistics on the predetermined statistical item according to the extracted log text. Get the statistics of the total amount of data transmission, the number of rows in the log text increment, the number of independent visitors, the number of successful requests, the number of failed requests, the reason for the failure of the request, and the number of times the data transmission was interrupted. Since the timing execution value of the trace log text increment can be freely set, the use of the cursor does not affect the content of the log text. The application can track the log text without intrusion, accurately extract the query result, and realize flexible log text. Processed efficiently.

Referring to FIG. 4, it is a flowchart of a second preferred embodiment of the log text processing method of the present application.

Compared with the first preferred embodiment, the present embodiment adds a text output control step, and the user can decide whether to output the extracted log text by controlling the text output switch. In this embodiment, the steps S10 to S40 are the same as the steps S10 to S40 in the first embodiment, and details are not described herein again.

Step S50, setting a text output switch to automatically determine whether the extracted log text meets a preset condition. The text output switch can be understood as a conditional branch instruction, and the preset condition can be various, such as status codes 408, 500, 504, etc., and the status codes indicate request timeout or server error, and the status codes are located. Additional information recorded in the log text line may help the operator to troubleshoot or improve server health. The preset condition may also be a range of data transmission bytes. When the number of data transmission bytes of the log record is within the range, the log text of the line satisfies a preset condition. For example, it is assumed that the preset condition is that the number of data transmission bytes is between 400 and 500 megabytes, and the log text line corresponding to the data transmission amount records information corresponding to the specific request sent by the client. It can be understood that the preset condition may be any condition that can be judged by the information recorded in the log text, and the purpose is to filter out a relatively more valuable part of the log text.

In step S60, the extracted log texts satisfying the preset condition are serialized and stored in the database. Since the data volume of the log is often very large, the utilization value of the extracted log text is also different. By setting the text output switch, the log text that satisfies the preset condition is automatically output, and serialized and stored in the database to improve the log text data. Security. Log text that does not meet the preset criteria remains in the original log file, but to conserve storage resources, these log text may be deleted after a period of time or overwritten by subsequent log text increments. It can be understood that the reason for serializing and storing the relatively valuable part of the log file, that is, the log text satisfying the preset condition, into the database is various, not only to save storage resources and improve log text data. safety. For example, ordinary files have advantages in sequential access, but they face many difficulties when files are large or randomly accessed, limiting access speed and extraction speed. In summary, the advantages of adding text output control steps during log text processing are conceivable.

In addition, the embodiment of the present application further provides a computer readable storage medium, which may be a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read only memory (ROM), and an erasable programmable Any combination or combination of any one or more of read only memory (EPROM), portable compact disk read only memory (CD-ROM), USB memory, and the like. The computer readable storage medium includes a log file, a log text processing program 10, and the like, and when the log text processing program 10 is executed by the processor 12, the following operations are implemented:

The specific implementation of the computer readable storage medium of the present application is substantially the same as the above-described log text processing method and the specific embodiment of the electronic device 1, and details are not described herein again.

It is to be understood that the term "comprises", "comprising", or any other variants thereof, is intended to encompass a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a series of elements includes those elements. It also includes other elements not explicitly listed, or elements that are inherent to such a process, device, item, or method. An element that is defined by the phrase "comprising a ..." does not exclude the presence of additional equivalent elements in the process, the device, the item, or the method that comprises the element.

Through the description of the above embodiments, those skilled in the art can clearly understand that the foregoing embodiment method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be through hardware, but in many cases, the former is better. Implementation. Based on such understanding, portions of the technical solution of the present application that contribute substantially or to the prior art may be embodied in the form of a software product stored in a storage medium as described above, including a number of instructions. To enable a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods described in the various embodiments of the present application.

The above is only a preferred embodiment of the present application, and is not intended to limit the scope of the patent application, and the equivalent structure or equivalent process transformations made by the specification and the drawings of the present application, or directly or indirectly applied to other related technical fields. The same is included in the scope of patent protection of this application.

Claims

A log text processing method is applied to an electronic device, the method comprising:

Text positioning step: obtain the log file path, open the log file, point the cursor to the last line of the current log text, and record the number of lines of the current log text as the cursor reference value;

Incremental tracking step: set the timing execution value, advance the cursor to the last line of the log text increment, and update the cursor reference value according to the number of rows of the log text increment;

Query extraction step: setting the query parameters, and using the cursor to extract the log text paragraphs satisfying the query parameters from the log text;

Statistical step: Statistics are performed on the extracted log text segments, and the statistical results of the predetermined statistical items are output.
The log text processing method according to claim 1, wherein in the text locating step, the cursor points to the first line of the current log text at the initial time, and is advanced line by line until pointing to the last line of the current log text. .
The log text processing method according to claim 1 or 2, wherein in the text locating step, the cursor is opened by using an OPEN statement, so that the cursor points to the first line of the current log text, and the cursor is advanced to the current state by using the FETCH statement. In the last line of the log text, use the CLOSE statement to close the cursor.
The log text processing method according to claim 1, wherein the amount of change in the cursor reference value is equal to the number of lines of the log text increment.
The log text processing method according to claim 1, wherein the query parameter comprises a time period and a keyword.
The log text processing method according to claim 1, wherein the statistical step is followed by a text output control step of: setting a text output switch, serializing the log text that satisfies the preset condition, and storing the log text in a database.
The log text processing method according to claim 1, wherein said predetermined statistical item comprises a total amount of data transmission and a number of lines of log text increment.
An electronic device includes a memory and a processor, wherein the memory includes a log text processing program, and when the log text processing program is executed by the processor, the following steps are implemented:

Text positioning step: obtain the log file path, open the log file, point the cursor to the last line of the current log text, and record the number of lines of the current log text as the cursor reference value;

Incremental tracking step: set the timing execution value, advance the cursor to the last line of the log text increment, and update the cursor reference value according to the number of rows of the log text increment;

Query extraction step: setting the query parameters, and using the cursor to extract the log text paragraphs satisfying the query parameters from the log text;

Statistical step: Statistics are performed on the extracted log text segments, and the statistical results of the predetermined statistical items are output.
The electronic device according to claim 8, wherein in the text locating step, the cursor initially points to the first line of the current log text, and is advanced line by line until it points to the last line of the current log text.
The electronic device according to claim 8 or 9, wherein in the text locating step, the cursor is opened by the OPEN statement, the cursor is pointed to the first line of the current log text, and the cursor is advanced to the current log text by using the FETCH statement. In the last line, use the CLOSE statement to close the cursor.
The electronic device of claim 8, wherein the amount of change in the cursor reference value is equal to the number of rows of the log text increment.
The electronic device of claim 8, wherein the query parameter comprises a time period and a keyword.
The electronic device according to claim 8, wherein the statistical step is followed by a text output control step of: setting a text output switch to serialize and store the log text that satisfies the preset condition in a database.
The electronic device of claim 8, wherein the predetermined statistical item comprises a total amount of data transmission and a number of lines of log text increments.
A computer readable storage medium, comprising: a log text processing program, wherein when the log text processing program is executed by a processor, the following steps are implemented:

Text positioning step: obtain the log file path, open the log file, point the cursor to the last line of the current log text, and record the number of lines of the current log text as the cursor reference value;

Incremental tracking step: set the timing execution value, advance the cursor to the last line of the log text increment, and update the cursor reference value according to the number of rows of the log text increment;

Query extraction step: setting the query parameters, and using the cursor to extract the log text paragraphs satisfying the query parameters from the log text; and

Statistical step: Statistics are performed on the extracted log text segments, and the statistical results of the predetermined statistical items are output.
The storage medium according to claim 15, wherein in said text locating step, said cursor initially points to a first line of the current log text, and is advanced line by line until it points to the last line of the current log text.
The storage medium of claim 15, wherein the amount of change in the cursor reference value is equal to the number of rows of the log text increment.
The storage medium of claim 15 wherein said query parameters comprise time periods and keywords.
The storage medium according to claim 15, wherein said statistical step is followed by a text output control step of: setting a text output switch to serialize and store the log text satisfying the preset condition in a database.
The storage medium of claim 15, wherein the predetermined statistical item comprises a total amount of data transmission and a number of lines of log text increment.