CN106777049B - Processing method and system for avoiding repeated log output - Google Patents
Processing method and system for avoiding repeated log output Download PDFInfo
- Publication number
- CN106777049B CN106777049B CN201611130080.8A CN201611130080A CN106777049B CN 106777049 B CN106777049 B CN 106777049B CN 201611130080 A CN201611130080 A CN 201611130080A CN 106777049 B CN106777049 B CN 106777049B
- Authority
- CN
- China
- Prior art keywords
- log
- information
- current
- file
- log information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/174—Redundancy elimination performed by the file system
- G06F16/1748—De-duplication implemented within the file system, e.g. based on file segments
Abstract
The invention discloses a processing method and a system for avoiding repeated log output, wherein the method comprises the following steps: s1, in the log recording system, calling a log writing interface function to transfer current log information, and adding a parameter unique identifier for the current log information; s2, generating and opening a new log file, acquiring a log list in a log recording system, comparing the current log information with the parameter unique identifier of the existing log information in the log list, and writing the current log information into the new log file if the same parameter unique identifier does not exist; if the same parameter unique identifier exists, the current log information is shown to exist in the existing log file, and the current log information is filtered; s3, closing the new log file and completing the processing of repeated log information. The invention avoids that a large amount of repeated messages fill the whole log file and also avoids that the log file is too bloated.
Description
Technical Field
The invention relates to the technical field of system log file processing, in particular to a processing method and a system for avoiding repeated log output.
Background
There may also be bugs that the program issues on the user's hands, the user may not remember what was done before or may be hard to describe the problem, it is often difficult for the program developer to fix these bugs, and the log can help the developer quickly locate the bug for the program. But sometimes it is found that the log file is very large and that there are a large number of repeated messages or messages that flood the entire log file, for example if not a fatal error repeat that would not result in the need to shut down the program.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a processing method and system for avoiding duplicate log output, aiming at the defects that log files are largely duplicated and a lot of resources are consumed by duplicate processing in the prior art.
The technical scheme adopted by the invention for solving the technical problems is as follows:
the invention provides a processing method for avoiding repeated log output, which comprises the following steps:
s1, in the log recording system, calling a log writing interface function to transfer current log information, and adding a parameter unique identifier for the current log information;
the log write interface function in step S1 is:
void log(int level，int code，const char*msg，...)
the level is the log level, msg is the log information, code is the unique parameter identifier, and the log parameters can be added at the end of the log writing interface function;
s2, generating and opening a new log file, acquiring a log list in a log recording system, comparing the current log information with the parameter unique identifier of the existing log information in the log list, and writing the current log information into the new log file if the same parameter unique identifier does not exist; if the same parameter unique identifier exists, the current log information is shown to exist in the existing log file, and the current log information is filtered;
the method for filtering the current log information in step S2 specifically includes:
judging whether current log information existing in the log list is expired:
if the log file is expired, writing current log information into the new log file;
if the current log information is not expired, updating the existing log file through the current log information, detecting the length of the current log information, and if the length of the current log information is greater than a set value of the log length, cutting off the current log information from the tail to enable the length of the current log information to meet the set value of the log length; if the length of the current log information is smaller than the set value of the log length, adding a blank space at the end of the current log information to enable the blank space to meet the set value of the log length;
when a log writing interface function is called to record log information msg, the related information of the msg log needs to be stored or updated, and an operation basis is provided for recording the msg next time; log related information is recorded through a structure stLogMsg and stLogFile;
the stLogMsg structure includes:
log identification id, last recording time nSecLastRecord of the log, position nSeekPos of the log record, log line nRow, log length nMoslen and recording time nRecordCnt of the log record;
the stLogFile structure includes:
the number of rows nRows of a log file, the maximum time interval nSecExceed between the latest record of a certain log and the last record, and the maximum number of rows interval nRowExceed between the latest record of a certain log and the last record;
s3, closing the new log file and completing the processing of repeated log information.
Further, when a new log file is generated in step S2 of the present invention, a name of the log file is generated by date, and the name of the new log file is added to the log list, in which the method includes:
generating the name of a log file through the current date, comparing the name with the name of the log file in the stLogMsg structural body, and if the name is the same as the name of the log file in the stLogMsg structural body, not processing the log file; and if the log file name is different, and the log file name is set to be a new log file name.
Further, the method for writing the log information in step S2 of the present invention specifically includes: and calling a fseek function to enable the position of the file pointer to point to the tail end of the log file, and writing log information into the log file by using an fputs function.
Further, in the method of the present invention, the method for judging whether the current log information existing in the log list is expired specifically includes:
and if the stLogFile:: nRows-stLogMsg:: nRow > -stLogFile:: nRowExced, the last log information corresponding to the current log information is expired.
Further, the parameters of the present invention are uniquely identified as log error codes.
The invention provides a processing system for avoiding repeated log output, which comprises:
the log writing interface function calling unit is used for calling a log writing interface function to transmit current log information in the log recording system and adding a parameter unique identifier for the current log information;
the log write interface function is:
void log(int level，int code，const char*msg，...)
the level is the log level, msg is the log information, code is the unique parameter identifier, and the log parameters can be added at the end of the log writing interface function;
the log information filtering unit is used for generating and opening a new log file, acquiring a log list in a log recording system, comparing the current log information with the parameter unique identifier of the existing log information in the log list, and writing the current log information into the new log file if the same parameter unique identifier does not exist; if the same parameter unique identifier exists, the current log information is shown to exist in the existing log file, and the current log information is filtered;
the method for filtering the current log information specifically comprises the following steps:
judging whether current log information existing in the log list is expired:
if the log file is expired, writing current log information into the new log file;
if the current log information is not expired, updating the existing log file through the current log information, detecting the length of the current log information, and if the length of the current log information is greater than a set value of the log length, cutting off the current log information from the tail to enable the length of the current log information to meet the set value of the log length; if the length of the current log information is smaller than the set value of the log length, adding a blank space at the end of the current log information to enable the blank space to meet the set value of the log length;
when a log writing interface function is called to record log information msg, the related information of the msg log needs to be stored or updated, and an operation basis is provided for recording the msg next time; log related information is recorded through a structure stLogMsg and stLogFile;
the stLogMsg structure includes:
log identification id, last recording time nSecLastRecord of the log, position nSeekPos of the log record, log line nRow, log length nMoslen and recording time nRecordCnt of the log record;
the stLogFile structure includes:
the number of rows nRows of a log file, the maximum time interval nSecExceed between the latest record of a certain log and the last record, and the maximum number of rows interval nRowExceed between the latest record of a certain log and the last record;
and the log output unit is used for closing the new log file and finishing the processing of repeated log information.
The invention has the following beneficial effects: according to the processing method for avoiding repeated log output, the purpose of identifying the log information is achieved by adding the unique parameter identification to the log information, and repeated log information can be effectively detected and filtered; the method can avoid that one or more repeated messages in large quantity fill the whole log file, and also avoid the problem that the log file is too fat because of the reason, thereby playing a certain role in quickly positioning the problem for developers and greatly improving the working efficiency.
Drawings
The invention will be further described with reference to the accompanying drawings and examples, in which:
FIG. 1 is a flow chart of an embodiment of the present invention;
fig. 2 is a diagram of a log-related structure according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As shown in fig. 1, the processing method for avoiding duplicate log output according to the embodiment of the present invention includes the following steps:
s1, in the log recording system, calling a log writing interface function to transfer current log information, and adding a parameter unique identifier for the current log information; the parameter of this embodiment is uniquely identified as a log error code.
The log write interface function is:
void log(int level，int code，const char*msg，...)
the level is the log level, msg is the log information, code is the unique parameter identification, and the log parameters can be added at the end of the log writing interface function.
When a log writing interface function is called to record log information msg, the related information of the msg log needs to be stored or updated, and an operation basis is provided for recording the msg next time; log related information is recorded through a structure stLogMsg and stLogFile;
the stLogMsg structure includes:
log identification id, last recording time nSecLastRecord of the log, position nSeekPos of the log record, log line nRow, log length nMoslen and recording time nRecordCnt of the log record;
the stLogFile structure includes:
the number of rows nRows of a log file, the maximum time interval nSecExceed between the latest record and the last record of a certain log, and the maximum row number interval nRowExceed between the latest record and the last record of a certain log.
S2, generating and opening a new log file, acquiring a log list in a log recording system, comparing the current log information with the parameter unique identifier of the existing log information in the log list, and writing the current log information into the new log file if the same parameter unique identifier does not exist; if the same parameter unique identifier exists, the current log information is shown to exist in the existing log file, and the current log information is filtered; the method for filtering the current log information specifically comprises the following steps:
judging whether current log information existing in the log list is expired:
if the log file is expired, writing current log information into the new log file;
if the current log information is not expired, updating the existing log file through the current log information, detecting the length of the current log information, and if the length of the current log information is greater than a set value of the log length, cutting off the current log information from the tail to enable the length of the current log information to meet the set value of the log length; if the length of the current log information is smaller than the set value of the log length, adding a blank space at the end of the current log information to enable the blank space to meet the set value of the log length.
S3, closing the new log file and completing the processing of repeated log information.
The processing system for avoiding repeated log output of the embodiment of the invention comprises:
the log writing interface function calling unit is used for calling a log writing interface function to transmit current log information in the log recording system and adding a parameter unique identifier for the current log information;
the log name updating unit is used for generating a new log file, and if the name of the new log file is different from the name of the existing log file in the log list, the name of the new log file is added into the log list;
the log information filtering unit is used for generating and opening a new log file, acquiring a log list in a log recording system, comparing the current log information with the parameter unique identifier of the existing log information in the log list, and writing the current log information into the new log file if the same parameter unique identifier does not exist; if the same parameter unique identifier exists, the current log information is shown to exist in the existing log file, and the current log information is filtered;
and the log output unit is used for closing the new log file and finishing the processing of repeated log information.
In another embodiment of the invention, the terms appearing are explained as follows:
·FILE
FILE is a C language structure actually defined by the system, and the structure contains information such as FILE name, FILE status and current location of the FILE. The details of the FILE structure need not be concerned when writing the source program. For example: FILE fp; the representation fp is a pointer variable pointing to a FILE structure, the structure variable storing certain FILE information can be found through fp, then the FILE is found according to the information provided by the structure variable, and the operation on the FILE is implemented. It is also customary to generically refer to fp as a pointer to a file, the file pointer pointing to a file in C language with a pointer variable.
In C language, file operations are all performed by library functions, the file operation functions involved in the present invention mainly include fread, fwrite, fseek, and ftell, fopen function is used to open a file,
the fclose function closes a file, fseek moves the read-write position of a file stream, and ftell is used for obtaining the current read-write position of the file.
Logs
Logs are information that records hardware, software, and system problems in a system, while also monitoring events that occur in the system. Through which the user can check the cause of the error or look for traces left by the attacker when under attack.
Introduction of related interfaces and functions:
·fopen
c language library function this, for opening files.
Function prototype: FILE fopen (const char path, const char mode);
parameters are as follows: "path" is the filename of the opened file; "mode" refers to the type and operating requirements of a file.
And returning a value: after the file is successfully opened, a file pointer to the stream is returned. If the file fails to open, NULL is returned and an error code is stored in errno.
·fclose
c language library function this, the function is to close a file stream. Note that: the last remaining data in the buffer can be output to the kernel buffer using the fclose () function and the file pointer and associated buffer are freed.
Function prototype: int fclose (FILE fp);
and returning a value: fclose returns 0 if the flow successfully closes, else EOF (-1) is returned. (if the stream is NULL and the program can continue, fclose sets error number to EINVAL and returns EOF.)
·fputs
The c-language library function has a function of writing a character string (not automatically writing the character string end marker '\ 0') to a specified file.
Function prototype: int fputs (char string, FILE fp);
parameters are as follows: string is the string to be written; fp is the file stream pointer.
And returning a value: after a character string is successfully written, the position pointer of the file can automatically move backwards, and the function return value is a non-negative integer; otherwise, EOF (sign constant, value-1) is returned.
·fseek
c language library function-the function of this function is to set the location of the file pointer.
Function prototype: int fseek (FILE fp, long offset, int fromwhere);
parameters are as follows: the parameter fp is an opened file pointer; the offset is the number of shifts to move the read/write position according to the fromwhere parameter. The parameter fromwhere is one of SEEK _ SET, SEEK _ CUR and SEEK _ END. SEEK _ SET indicates that the offset displacement from the beginning of the file is the new read/write position, SEEK _ CUR increases the offset displacement backwards from the current read/write position, and SEEK _ END points the read/write position to the END of the file and then increases the offset displacement.
And returning a value: and returning to 0 if the calling is successful, returning to-1 if an error exists, and storing an error code by the errno.
·ftell
c language library function, which is used to obtain the current reading and writing position of the file stream.
Function prototype: long ftell (FILE × fd);
and returning a value: and returning to the current read-write position when the calling is successful, returning to-1 if an error exists, and storing an error code by the errno.
A general logging system will have a log writing interface function, and parameters of the function must have log content, and other parameters such as log level may be selected, for example, void log (int level, con char msg.) the function parameters include log level and log content. The invention needs to avoid the situation that the same log is repeatedly output to a file in a short time, so that the log message needs to be uniquely confirmed, the log function can be newly filled with a parameter unique mark log, and the hash value of the msg parameter can be calculated in the log writing interface function to determine that the msg parameter is one. Since the log writing interface function used by the system developed by the inventor needs to transmit a log error code, the parameter is used as the only indication of the log. The log function is void log (int level, int code, constchar × msg.).
When the call log is written into the interface function record msg, the related information of the msg log needs to be saved or updated, and the judgment is made on how to operate when the msg is recorded next time. The log-related structure is shown in FIG. 2:
the stLogMsg structural body records information of a certain type of log, and the log is uniquely identified through an error code, so that the comments of stLogMsg structural body can be seen through introduction of more members, wherein id is the error code.
The stLogFile is a log file structure, stLogFile:: nSecExended and stLogFile:: nRowExended are judgment conditions for log information expiration, if the last recorded information of the log is expired, the current log needs to be processed as a new log, and the new log needs to be recorded at the current tail position of the file. When the interval between the current record and the last record of a certain type of log is too long or the interval between the log and the row of the latest log of the log file is too far, the log is called as log expiration. strFileName is the name of the log file, and when the log file changes, all log information of the stLogMsg:lstLogrepresenting the stLogMsg structural object of the file needs to be cleaned.
The following describes in detail the log record implementation procedure in this embodiment (before the log is recorded by calling the log write interface function, log information expiration conditions such as stLogFile:: nSecExceed and stLogFile:: nRowExceed need to be set, stLogFile:: nSecExceed is 3, stLogFile:: nRowExceed is 50) set by this system:
a) and calling a log writing interface function, and transmitting parameters such as log level, error codes, msg content and the like.
b) And generating the name of a log file through the current date (the system requires one log every day, if the system runs across days, a new log file is used, and the log file is named according to the date), comparing with the stLogMsg:: strFileName, if the log files are different (the log files are the same, no processing is performed), indicating that the log files are different, clearing all log information stored in the stLogMsg:: lstLog, and setting a new log file name for the stLogMsg:: strFileName.
c) And opening a log file by the fopen function, writing an interface function parameter code into the log, and searching whether log record information exists in the stLogMsg:: lstLog. If not, executing step d, otherwise executing step e.
d) Call fseek (fd,0L, SEEK _ END) to point the location of the file pointer to the END of the file, write the msg contents to the file using the fputs function, and increase the number of lines of the file, stlogFile:: nRows, by 1 (note that each log is written in a line in the file).
Since this msg is written to the log file record for the first time, it is necessary to add log information of the stLogMsg structure to the stLogMsg:: lstLog.
stLogMsg: : the id is assigned as the code and,
stLogMsg nSecLastRecord is obtained by obtaining system time,
and the stLogMsg is the file writing position of the current log, the new log needs to be recorded to the current tail position of the file, fseek (fd,0L, SEEK _ END) needs to be called to obtain the writing position of the current log, the position of a file pointer points to the tail of the file, and then the current position of the file is obtained through the ftell.
nRow is the row of the file where the log is located, and the assignment is STLogFile:: nRows;
nRecordCnt the log recording times are set as 1 due to the first recording;
stLogMsg:: nMsgLen length of the log content (note that information of the number of log records needs to be added here, which is information forcibly added by the present log system after log information, although repeated log information is not recorded, the number of execution times of the log can be known through information of the number of log records).
e) Since the msg already has log information in the stLogMsg:: lstLog, it is decided how to process the piece of information by the stLogMsg structure log information. First, whether the piece of log information is expired is determined. If the stLogFile:: nRows-stLogMsg:: nRow > -stLogFile:: nRowExceed or system time stLogMsg:: nRow-nSecLastRecord:: stLogFile:: nRowExceed, it indicates that the last log recording information of the current msg is expired (otherwise, it is not expired), and the msg needs to be recorded to the end of the file as a new log, and the operation is the same as that in step d.
It is also a particular aspect of the present invention that if the log information is not expired, only the last log needs to be updated by the msg. The log updating steps are as follows:
check the length of the current msg (typically the log lengths of the same code identifiers are the same, but may differ for some reason, as exemplified below.) additionally, because the writing of the file is an overwrite, if the length of the current msg is greater than the last time, it may overwrite other logs),
i. nMsgLen if the length of the current msg > stLogMsg: (note that the length of the log is all
Is the length after forced addition of fixed length information, so the length of msg needs to be added to this length as well), truncating the msg from the end makes the length of msg still stLogMsg:nmsglen (the log can also be treated as expired),
if the current msg length < stLogMsg:: nMsgLen, then add null after msg
The length of the grid filling from msg to msg is stLogMsg:nMSgLen.
For example: there is a log of open camera failure that has been written to file for the first time, the log may have different log lengths due to different camera names (note that the string following the log '}' is information of the number of log records forcibly added, the length of the information is 18, the times string colon is followed by 8 spaces stLogMsg:: nRecordCnt, i.e. the number of log records stLogMsg:: nRecordCnt is stored with a length of 9 characters)
[2016-XX-XX 18:48:06.288][ERROR][0xFFFF0001]{open camera CAMERA10 failed.}-times:1
After the log is written into a file, log information stLogMsg is updated, wherein nMsgLen is 94 (the length of msg is 76+ 18);
if now there is another msg of the type that fails to open the CAMERA (this msg is [2016-XX-XX18:48:06.688] [ ERROR ] [0xFFFF0001] { open CAMERA CAMERA100failed. }) to write the journal, because the length of this msg length 77 plus the information length 18 of the number of journal recordings must be greater than stLogMsg:: nMsgLen, after truncation of this msg, this msg is [2016-XX-XX18:48:06.688] [ ERROR ] [0xFFFF0001] { open CAMERA CAMERA100 failed).
The log content of the written file is:
[2016-XX-XX 18:48:06.688][ERROR][0xFFFF0001]{open camera CAMERA100 failed.-times:2
then there is a log to be written with a msg of the type that failed to turn on the CAMERA (this msg is [2016-XX-XX18:48:06.688] [ ERROR ] [0xFFFF0001] { open CAMERA Camera1 failed. }), which can be seen to be 75 in length, and after the space is filled, this msg is (this msg is followed by a space):
[2016-XX-XX 18:48:06.888][ERROR][0xFFFF0001]{open camera CAMERA1 failed.}
the log content of the written file is:
[2016-XX-XX 18:48:06.888][ERROR][0xFFFF0001]{open camera CAMERA1 failed.}-times:3
and (4) increasing the log recording times stLogMsg:nRecordCntby 1 and updating the log recording times stLogMsg into the log information. By updating the number of times of the logs without repeatedly recording the logs, information such as the number of logs is not lost, and the repetition is avoided.
And calling the fseek function to move the file pointer to the position of the stLogMsg, nSeekPos, calling the fputs function to write the log character string into the file, and ending the updating.
f) The log file is closed using the fclose function.
It can be seen that, with this scheme, when there is a large amount of duplicate identical log output, only the number information of log records will be updated on the basis of the log that has been recorded before, and no large amount of duplicate log information will appear in the log file. And because the conditions of setting the log information to be overdue such as the stLogFile:nSecExcedand the stLogFile:nRowExceddo not cause the latest written repeated log to be too far away from the tail position of the log file. The scheme avoids that a large number of repeated messages or a plurality of repeated messages fill the whole log file, and also avoids the problem that the log file is too bloated due to the reason, thereby playing a certain role in quickly positioning the problem for developers.
It will be understood that modifications and variations can be made by persons skilled in the art in light of the above teachings and all such modifications and variations are intended to be included within the scope of the invention as defined in the appended claims.
Claims (6)
1. A processing method for avoiding duplicate log output is characterized by comprising the following steps:
s1, in the log recording system, calling a log writing interface function to transfer current log information, and adding a parameter unique identifier for the current log information;
the log write interface function in step S1 is:
defining a log writing interface function, wherein input parameters of the function comprise: the log grade, the log information and the unique parameter identification are adopted, and log parameters can be added at the tail of a log writing interface function;
s2, generating and opening a new log file, acquiring a log list in a log recording system, comparing the current log information with the parameter unique identifier of the existing log information in the log list, and writing the current log information into the new log file if the same parameter unique identifier does not exist; if the same parameter unique identifier exists, the current log information is shown to exist in the existing log file, and the current log information is filtered;
the method for filtering the current log information in step S2 specifically includes:
judging whether current log information existing in the log list is expired:
if the log file is expired, writing current log information into the new log file;
if the current log information is not expired, updating the existing log file through the current log information, detecting the length of the current log information, and if the length of the current log information is greater than a set value of the log length, cutting off the current log information from the tail to enable the length of the current log information to meet the set value of the log length; if the length of the current log information is smaller than the set value of the log length, adding a blank space at the end of the current log information to enable the blank space to meet the set value of the log length;
when a log writing interface function is called to record log information msg, the related information of the msg log needs to be stored or updated, and an operation basis is provided for recording the msg next time; log related information is recorded through a structure stLogMsg and stLogFile;
the stLogMsg structure includes:
log identification id, last recording time nSecLastRecord of the log, position nSeekPos of the log record, log line nRow, log length nMoslen and recording time nRecordCnt of the log record;
the stLogFile structure includes:
the number of rows nRows of a log file, the maximum time interval nSecExceed between the latest record of a certain log and the last record, and the maximum number of rows interval nRowExceed between the latest record of a certain log and the last record;
s3, closing the new log file and completing the processing of repeated log information.
2. The method of claim 1, wherein when a new log file is generated in step S2, the name of the log file is generated according to the date, and the name of the new log file is added to the log list, and the method comprises:
generating the name of a log file through the current date, comparing the name with the name of the log file in the stLogMsg structural body, and if the name is the same as the name of the log file in the stLogMsg structural body, not processing the log file; and if the log file name is different, and the log file name is set to be a new log file name.
3. The processing method for avoiding duplicate log output according to claim 1, wherein the method for writing the log information in step S2 specifically comprises: and calling a fseek function to enable the position of the file pointer to point to the tail end of the log file, and writing log information into the log file by using an fputs function.
4. The processing method for avoiding duplicate log output according to claim 1, wherein the method for determining whether current log information existing in the log list is expired specifically comprises:
and if the stLogFile:: nRows-stLogMsg:: nRow > -stLogFile:: nRowExced, the last log information corresponding to the current log information is expired.
5. A processing method for avoiding duplicate log output according to claim 1, wherein the parameter in the method is uniquely identified as a log error code.
6. A processing system for avoiding duplicate log output, comprising:
the log writing interface function calling unit is used for calling a log writing interface function to transmit current log information in the log recording system and adding a parameter unique identifier for the current log information;
the log write interface function is:
defining a log writing interface function, wherein input parameters of the function comprise: the log grade, the log information and the unique parameter identification are adopted, and log parameters can be added at the tail of a log writing interface function;
the log information filtering unit is used for generating and opening a new log file, acquiring a log list in a log recording system, comparing the current log information with the parameter unique identifier of the existing log information in the log list, and writing the current log information into the new log file if the same parameter unique identifier does not exist; if the same parameter unique identifier exists, the current log information is shown to exist in the existing log file, and the current log information is filtered;
the method for filtering the current log information specifically comprises the following steps:
judging whether current log information existing in the log list is expired:
if the log file is expired, writing current log information into the new log file;
if the current log information is not expired, updating the existing log file through the current log information, detecting the length of the current log information, and if the length of the current log information is greater than a set value of the log length, cutting off the current log information from the tail to enable the length of the current log information to meet the set value of the log length; if the length of the current log information is smaller than the set value of the log length, adding a blank space at the end of the current log information to enable the blank space to meet the set value of the log length;
when a log writing interface function is called to record log information msg, the related information of the msg log needs to be stored or updated, and an operation basis is provided for recording the msg next time; log related information is recorded through a structure stLogMsg and stLogFile;
the stLogMsg structure includes:
log identification id, last recording time nSecLastRecord of the log, position nSeekPos of the log record, log line nRow, log length nMoslen and recording time nRecordCnt of the log record;
the stLogFile structure includes:
the number of rows nRows of a log file, the maximum time interval nSecExceed between the latest record of a certain log and the last record, and the maximum number of rows interval nRowExceed between the latest record of a certain log and the last record;
and the log output unit is used for closing the new log file and finishing the processing of repeated log information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611130080.8A CN106777049B (en) | 2016-12-09 | 2016-12-09 | Processing method and system for avoiding repeated log output |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611130080.8A CN106777049B (en) | 2016-12-09 | 2016-12-09 | Processing method and system for avoiding repeated log output |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106777049A CN106777049A (en) | 2017-05-31 |
| CN106777049B true CN106777049B (en) | 2021-01-01 |
Family
ID=58875729
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611130080.8A Active CN106777049B (en) | 2016-12-09 | 2016-12-09 | Processing method and system for avoiding repeated log output |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106777049B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019064A (en) * | 2017-09-01 | 2019-07-16 | 大唐移动通信设备有限公司 | Eliminate the filter method and device for repeating log recording |
| CN108427619B (en) * | 2018-03-13 | 2020-10-20 | 腾讯科技（深圳）有限公司 | Log management method and device, computing equipment and storage medium |
| CN108563718B (en) * | 2018-04-02 | 2021-07-23 | 郑州云海信息技术有限公司 | Method and system for preventing log flood |
| CN108829543A (en) * | 2018-06-21 | 2018-11-16 | 郑州云海信息技术有限公司 | A method of reducing backup Linux system log size |
| CN110515803B (en) * | 2019-08-27 | 2021-04-13 | 联想(北京)有限公司 | Processing method and device for log message and electronic equipment |
| CN110908972B (en) * | 2019-11-19 | 2022-09-02 | 加和（北京）信息科技有限公司 | Log data preprocessing method and device, electronic equipment and storage medium |
| CN111967850B (en) * | 2020-08-19 | 2022-10-18 | 支付宝(杭州)信息技术有限公司 | Data reporting monitoring method and device and electronic equipment |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7502447B2 (en) * | 2003-11-25 | 2009-03-10 | Alcatel Lucent | Call failure recording |
| CN102880718B (en) * | 2012-10-12 | 2015-09-02 | 互动在线（北京）科技有限公司 | A kind of storage of flexible daily record and acquisition methods |
| CN104156420B (en) * | 2014-08-06 | 2017-10-03 | 曙光信息产业(北京)有限公司 | The management method and device of transaction journal |
| CN104915460A (en) * | 2015-07-10 | 2015-09-16 | 上海斐讯数据通信技术有限公司 | Log storage method and system |
| CN105049260B (en) * | 2015-08-24 | 2019-03-19 | 浪潮(北京)电子信息产业有限公司 | Blog management method and device |
| CN105955972A (en) * | 2016-02-01 | 2016-09-21 | 上海华测导航技术股份有限公司 | Receiver log management system and method |
-
2016
- 2016-12-09 CN CN201611130080.8A patent/CN106777049B/en active Active
Non-Patent Citations (2)
| Title |
|---|
| Heejin Park;Sang-Chul Lee;Soon-Haeng Lee;Sang-Wook Kim.CentralMatch: A Fast and Accurate Method to Identify Blog-Duplicates.《2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology》.2010, * |
| 一种能发现重复任务的过程挖掘算法;李嘉菲等;《吉林大学学报（工学版）》;20070131;第106-110页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106777049A (en) | 2017-05-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106777049B (en) | Processing method and system for avoiding repeated log output | |
| CN107391628B (en) | Data synchronization method and device | |
| KR101805948B1 (en) | Checkpoints for a file system | |
| CN107515874B (en) | Method and equipment for synchronizing incremental data in distributed non-relational database | |
| US20150213100A1 (en) | Data synchronization method and system | |
| CN110659256B (en) | Multi-computer room synchronization method, computing device and computer storage medium | |
| CN102999433B (en) | Redundant data deletion method and system of virtual disks | |
| CN109710185A (en) | Data processing method and device | |
| CN111400267A (en) | Method and device for recording log | |
| CN113704790A (en) | Abnormal log information summarizing method and computer equipment | |
| US20070192384A1 (en) | Computer implemented method for automatically managing stored checkpoint data | |
| CN110737504B (en) | Fault-tolerant method, system, terminal and storage medium for deep learning model training | |
| CN109284331B (en) | Certificate making information acquisition method based on service data resources, terminal equipment and medium | |
| CN109446262A (en) | A kind of data assemblage method and device | |
| CN112988880B (en) | Data synchronization method, device, electronic equipment and computer storage medium | |
| CN111078418B (en) | Operation synchronization method, device, electronic equipment and computer readable storage medium | |
| WO2017067397A1 (en) | Data recovery method and device | |
| CN112000623A (en) | Metadata access method and device and computer readable storage medium | |
| CN111026764B (en) | Data storage method and device, electronic product and storage medium | |
| CN115712397A (en) | Cache verification device, method and system | |
| CN111460436B (en) | Unstructured data operation method and system based on blockchain | |
| CN115455059A (en) | Method, device and related medium for analyzing user behavior based on underlying data | |
| CN112650613B (en) | Error information processing method and device, electronic equipment and storage medium | |
| CN114356404A (en) | Interface document generating method, system and computer readable storage medium | |
| CN108614868B (en) | Automatic database upgrading method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |