CN112231279A

CN112231279A - Log management system

Info

Publication number: CN112231279A
Application number: CN202011093681.2A
Authority: CN
Inventors: 张亚; 刘彦; 黄风雷
Original assignee: Chongqing Innovation Center of Beijing University of Technology
Current assignee: Chongqing Innovation Center of Beijing University of Technology
Priority date: 2020-10-14
Filing date: 2020-10-14
Publication date: 2021-01-15
Anticipated expiration: 2040-10-14
Also published as: CN112231279B

Abstract

The invention provides a log management system, which comprises a log acquisition module, a log filing module, a log cleaning module and a log exporting module, wherein the log management system classifies logs according to log types through the log filing module and compresses and packs the logs according to time, so that a more exquisite filing method is provided, and accurate positioning and quick searching of the logs are facilitated; the system also provides an automatic log cleaning function, a user can define a log cleaning size threshold and a log cleaning time threshold in a self-defined mode, unnecessary logs are cleaned automatically under the condition that the log data volume is too large or the storage time is too long, the occupied space is reduced, the utilization rate of the storage space is improved, and the stability of the computer system is ensured; in addition, the system can automatically search and export the log to the designated path based on the log type and the log generation time period, provides a simple, convenient and quick log acquisition mode for the user, and is favorable for providing the utilization rate of the log.

Description

Log management system

Technical Field

The invention relates to the technical field of computers, in particular to a log management system.

Background

The Log (Log) refers to a collection of operations of the specified objects of the system and the operation results thereof in order of time. Each log file is made up of log records, each describing a separate system event. Typically, the system log is a text file that the user can read directly, containing a timestamp and an information or other information specific to the subsystem.

The log file records necessary and valuable information for IT resource related activities such as servers, workstations, firewalls, application software and the like, which is very important for system monitoring, inquiry, reporting and security audit. Records in the log file may provide monitoring system resources; auditing user behavior; alarming suspicious behaviors; determining the scope of the intrusion behavior; help is provided for restoring the system; generating a survey report; providing evidence sources for fighting against computer crimes, and the like.

The capacity generated by one log file in one day is few tens of megabytes and hundreds of megabytes, more than a few G, and dozens of G, which are all stored directly according to the time sequence without effective processing, so that the redundancy is very large, the useful information is difficult to search and analyze, and the utilization rate of the computer log file is low at present.

Disclosure of Invention

The invention provides a log management system, which mainly solves the technical problems that: how to facilitate the lookup and analytical use of computer log files/records.

To solve the above technical problem, the present invention provides a log management system, including:

a log obtaining module, a log filing module, a log cleaning module and a log exporting module, wherein,

the log obtaining module comprises N obtaining sub-modules, wherein one obtaining sub-module is used for correspondingly obtaining one type of log files/records, and N is more than or equal to 2;

the log archiving module comprises a first processing submodule, a cache unit, a first configuration submodule, a packaging submodule and a storage unit; n buffer queues are arranged in the buffer unit; the first processing sub-module is used for putting the acquired log files/records into corresponding cache queues, and one type of log file/record is correspondingly provided with one cache queue; the first configuration submodule is used for receiving an external first setting instruction so as to set a packaging period; the packing submodule is used for compressing and packing the log files/records in the cache queue when the judgment time reaches the packing period to obtain a compressed packet of the log files/records, and emptying the cache queue; the storage unit is used for storing the compressed packet;

the log cleaning module comprises a second processing submodule, a second configuration submodule and a third configuration submodule; the second configuration submodule is used for receiving an external second setting instruction so as to set a log clearing size threshold; the third configuration submodule is used for receiving an external third setting instruction so as to set a log clearing time threshold; the second processing submodule is used for sequentially selecting at least part of compressed packets of the log files/records to delete according to the sequence of the log generation time when judging that the size of the log files/records stored in the storage unit exceeds the log clearing size threshold or the log generation time exceeds the log clearing time threshold;

the log export module comprises a first instruction receiving submodule, a second instruction receiving submodule, a third instruction receiving submodule and an export submodule; the first instruction receiving submodule is used for receiving the type of the export log; the second instruction receiving submodule is used for receiving a generation time period of the export log; the third instruction receiving submodule is used for receiving a storage path of an export log; and the export submodule is used for searching the matched target compressed packet in the storage unit according to the type of the export log and/or the generation time period and exporting the matched target compressed packet to the storage path.

Further, the types of the log files/records include a user management log, a system log, a configuration modification log, a command downlink log, a data uplink log, a performance monitoring log, and a security log.

The invention has the beneficial effects that:

the log management system comprises a log acquisition module, a log filing module, a log cleaning module and a log exporting module, and the log management system classifies logs according to log types through the log filing module and compresses and packs the logs according to time, so that a more exquisite filing method is provided, and accurate positioning and quick searching of the logs are facilitated; the system also provides an automatic log cleaning function, a user can define a log cleaning size threshold and a log cleaning time threshold in a self-defined mode, unnecessary logs are cleaned automatically under the condition that the log data volume is too large or the storage time is too long, the occupied space is reduced, the utilization rate of the storage space is improved, and the stability of the computer system is ensured; in addition, the system can automatically search and export the log to the designated path based on the log type and the log generation time period, provides a simple, convenient and quick log acquisition mode for the user, and is favorable for providing the utilization rate of the log.

Drawings

FIG. 1 is a schematic diagram of a log management system according to the present invention;

FIG. 2 is a schematic diagram of a log archiving process according to the present invention;

FIG. 3 is a schematic diagram illustrating an automatic log cleaning process according to the present invention;

fig. 4 is a schematic diagram of a log export process according to the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following detailed description and accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Referring to fig. 1, fig. 1 is a schematic structural diagram of a log management system of the present invention, the system includes a log obtaining module 10, a log archiving module 20, a log cleaning module 30, and a log exporting module 40, wherein:

the log obtaining module 10 includes N obtaining sub-modules 11, where one obtaining sub-module 11 is used to correspondingly obtain one type of log file/record, and N is greater than or equal to 2. The log types can be divided according to the service types, and one service type corresponds to one log type. In this embodiment, the log types specifically include the following: user management logs, system logs, configuration modification logs, command downlink logs, data uplink logs, performance monitoring logs and security logs. It should be understood that the above-mentioned log types are only alternative embodiments of the present invention, and the present invention is not limited to the above-mentioned log types, nor to the above-mentioned division manner. The invention is used for carrying out classified acquisition, archiving and exporting on the logs, is convenient for management, analysis and use of the logs and provides the effective utilization rate of the logs. Therefore, it may include various different log types, or different log types obtained by dividing according to different dividing manners (e.g. service types).

The log archiving module 20 comprises a first processing submodule 21, a cache unit 22, a first configuration submodule 23, a packing submodule 24 and a storage unit 25; wherein, N buffer queues are arranged in the buffer unit 22; the first processing sub-module 21 is configured to put the obtained log file/record into a corresponding cache queue, where a cache queue is correspondingly provided for one type of log file/record; the first configuration submodule 23 is configured to receive an external first setting instruction to set a packing period; the packing submodule 24 is configured to, when it is determined that the time reaches the packing period, perform compression packing on the log files/records in the cache queue to obtain a compressed packet of the log files/records, and empty the cache queue; the storage unit 25 is configured to store the compressed packet obtained by the compression and packaging processing of the packaging submodule 24. A more exquisite filing method is provided, and accurate positioning and quick searching of the log are facilitated.

The log archiving process of the log archiving module 20 is shown in fig. 2, and includes:

s201, putting the acquired log file/record into a corresponding cache queue;

s202, judging whether the time exceeds a packaging period; if yes, go to step S203; if not, monitoring a newly acquired log file/record;

and S203, compressing and packaging to obtain a compressed package.

The obtaining sub-module 11 is configured to correspondingly obtain one type of log file/record, where the one type of log file/record is correspondingly provided with a cache queue, and place the obtained log file/record into the corresponding cache queue through the first processing sub-module 21, so as to implement automatic archiving. The purpose of automatically identifying the log type can be achieved by establishing the mapping relation among the acquisition submodule 11, the log type and the cache queue without setting a log type identification module or an identification algorithm.

As a preferred mode of the present invention, the packing sub-module 24 may be further configured to, after the determination time reaches the packing period, before performing compression packing on the log files/records in the cache queue, determine whether the cache queue is empty, if so, not perform packing processing, and avoid generating an empty compression packet; on the contrary, if the buffer queue is not empty, that is, there is a log file/record, the compression and packing process is performed.

The log cleaning module 30 comprises a second processing submodule 31, a second configuration submodule 32 and a third configuration submodule 33; the second configuration submodule 32 is configured to receive an external second setting instruction to set a log cleaning size threshold; the third configuration submodule 33 is configured to receive an external third setting instruction to set a log clearing time threshold; the second processing submodule 31 is configured to, when it is determined that the size of the log file/record stored in the storage unit 25 exceeds the log clearing size threshold or the log generation time exceeds the log clearing time threshold, sequentially select at least some compressed packets of the log file/record according to the sequence of the log generation time to delete the compressed packets; the space occupation is reduced, the utilization rate of the storage space is improved, and the stability of the computer system is ensured.

The log cleaning process of the log cleaning module 30 is shown in fig. 3, and includes:

s301, judging that the size of the log file/record stored in the storage unit exceeds the log clearing size threshold; if yes, go to step S303; if not, go to step S302;

s302, judging whether the log generation time exceeds the log clearing time threshold; if yes, go to step S303; if not, ending;

it should be understood that, in an alternative embodiment of the present invention, step S302 may be executed first, and then step S301 may be executed, which is not limited by the present invention and can be flexibly processed.

And S303, sequentially selecting at least part of the log files/recorded compressed packets according to the sequence of the log generation time to delete.

It should be understood that the log cleaning size threshold can be flexibly set according to the size of the computer storage space or the actual requirement for the log file, and the scheme does not limit the specific log cleaning size threshold; similarly, the log clearing time threshold can also be flexibly set according to the size of the storage space of the computer or the actual requirement on the log file, and longer log files/records can be stored if the log clearing time threshold is larger, and conversely, log files/records generated in a short period of time can only be stored if the log clearing time threshold is smaller. The log clean-up time threshold is set to, for example, 3 months, i.e., the system can save log files/records for nearly 3 months.

In this embodiment, when it is determined that the size of the log file/record stored in the storage unit 25 exceeds the log clearing size threshold and the log generation time exceeds the log clearing time threshold, at least a part of the compressed packets of the log file/record may be selected for deletion as follows:

1. and deleting according to the set data size. For example, the log cleaning size threshold is 100G, and the set data amount should be smaller than the log cleaning size threshold, for example, the set data amount is set to 10% of the log cleaning size threshold, that is, 10G; and sequentially selecting log files/records with the earlier time according to the sequence of the log generation time, deleting the compressed packets with the data volume of about 10G, and retaining the log files/records with the later time of 90G.

2. And deleting according to the set time days. For example, the log cleaning time threshold is set to 90 days, and the set time days should be less than the log cleaning time threshold, for example, set to 10 days here; when the size of the log file/record stored in the storage unit 25 is judged to exceed the log clearing size threshold and the log generation time exceeds the log clearing time threshold, the log file/record within 10 days before the most time is selected for deletion according to the sequence of the log generation time, and the log file/record within 90 days after the deletion is reserved.

3. The log file/record that has been exported is first deleted, and the deletion is performed in the above-described manner 1 or manner 2. Generally, the exported log files/records are backed up by a user, so that the exported log files/records are firstly deleted, on one hand, the acquisition and use of the log files/records by the user are not influenced, on the other hand, the system can store more log files/records, and the data traceability is improved.

The log export module 40 comprises a first instruction receiving submodule 41, a second instruction receiving submodule 42, a third instruction receiving submodule 43 and an export submodule 44; wherein, the first instruction receiving submodule 41 is configured to receive a type of the export log; the second instruction receiving submodule 42 is configured to receive a generation time period of the export log; the third instruction receiving submodule 43 is configured to receive a storage path of the export log; the export submodule 44 is configured to search the storage unit 25 for a matching target compressed packet according to the type and/or the generation time period of the export log, and export the target compressed packet to the storage path. A simple, convenient and quick log obtaining mode is provided for a user, and the log utilization rate is favorably improved.

The log export process of the log export module 40 is shown in fig. 4, and includes:

s401, acquiring the type of a log required to be exported;

s402, acquiring a generation time period of a log needing to be exported;

s403, acquiring a derivation path;

s404, according to the type and/or the generation time period of the export log, the matched target compressed packet is searched in the storage unit 25 and exported to the storage path.

It should be understood that if the user does not input the type of export log, the system may search the storage unit 25 for the matching target compressed packet according to the generation time period of the desired export log and export the target compressed packet; if the user does not input the generation time period of the export log, the system may search the storage unit 25 for the matching target compressed packet according to the type of the desired export log, and export the target compressed packet.

It will be apparent to those skilled in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be centralized on a single computing device or distributed across a network of computing devices, and optionally they may be implemented in program code executable by a computing device, such that they may be stored on a computer storage medium (ROM/RAM, magnetic disks, optical disks) and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different than that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

The foregoing is a more detailed description of the present invention that is presented in conjunction with specific embodiments, and the practice of the invention is not to be considered limited to those descriptions. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.

Claims

1. The log management system is characterized by comprising a log acquisition module, a log archiving module, a log cleaning module and a log exporting module,

2. The log management system of claim 1, wherein the types of log files/records include a user management log, a system log, a configuration modification log, a command downline log, a data upline log, a performance monitoring log, and a security log.