CN117176441A - System and method for detecting security log event of network equipment - Google Patents

System and method for detecting security log event of network equipment Download PDF

Info

Publication number
CN117176441A
CN117176441A CN202311189838.5A CN202311189838A CN117176441A CN 117176441 A CN117176441 A CN 117176441A CN 202311189838 A CN202311189838 A CN 202311189838A CN 117176441 A CN117176441 A CN 117176441A
Authority
CN
China
Prior art keywords
data
log
event
security
analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311189838.5A
Other languages
Chinese (zh)
Inventor
步敬敏
王国广
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Jiahong Information Technology Co ltd
Original Assignee
Shanghai Jiahong Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Jiahong Information Technology Co ltd filed Critical Shanghai Jiahong Information Technology Co ltd
Priority to CN202311189838.5A priority Critical patent/CN117176441A/en
Publication of CN117176441A publication Critical patent/CN117176441A/en
Pending legal-status Critical Current

Links

Landscapes

  • Debugging And Monitoring (AREA)

Abstract

The invention discloses a system and a method for detecting a security log event of network equipment. The invention develops a safe log detection system capable of completely meeting specific network requirements, can be applied to different kinds of network equipment, effectively solves the problem of network equipment variability, so as to meet the requirements of all users on log collection and detection, greatly reduces the workload of operation and maintenance personnel by introducing artificial intelligence into log anomaly detection, and can pre-process log data before log anomaly detection so as to ensure that too many errors and irrelevant data are not introduced in the whole process from data acquisition, storage until analysis and visualization, and can arrange the data according to a certain sequence, thereby effectively improving the accuracy and reducing the false alarm rate.

Description

System and method for detecting security log event of network equipment
Technical Field
The invention belongs to the technical field of network equipment security, and particularly relates to a system and a method for detecting a security log event of network equipment.
Background
With the development and application of information technology, networks have become an integral part of people's daily life and work, and network security has become more and more important. In order to ensure the security of information systems, web security logs are an indispensable tool. The network security log is a file for recording various security events and operation records of the network equipment system, can record information such as access conditions, login records, attack behaviors, system operation, abnormal conditions and the like of the website equipment, and has important functions for detecting, analyzing and solving the network security events. The network security log is an active defense means based on a log recording technology, and can help security personnel to discover and check security holes in time and prevent and deal with network security events in time.
The network security log records a large amount of data in the network operation process, and can timely discover, check and process network security events by analyzing and detecting log contents, so that the network security is improved, but in actual work, the analysis and detection of the network security log has some problems: (1) The network devices on the market are various in types, the device difference is large, the requirements of all users on network security log collection and detection cannot be met, and a security log detection system capable of completely meeting the requirements of a specific network is absent; (2) The network security log belongs to semi-structured data, and once log abnormality occurs, operation and maintenance personnel need to search for the abnormality in a large amount of log data, which is a work with huge workload and very high cost, and the detection work of the network security log event is difficult to be completed quickly and timely by relying on manpower alone; (3) The existing network security log abnormality detection algorithm mainly adopts a time sequence-based method to detect abnormality, but in practice, log index time sequence data does not have periodicity or has other distribution characteristics, and abnormality detection is carried out only according to periodicity, so that the problems of high false alarm rate, low accuracy and the like are caused. Therefore, aiming at the problems, the system and the method for detecting the security log event of the network equipment have important significance.
Disclosure of Invention
The invention provides a system and a method for detecting security log events of network equipment, which can be applied to different kinds of network equipment by developing a security log detection system capable of completely meeting specific network requirements, and effectively solve the problem of the variability of the network equipment so as to meet the requirements of all users on log collection and detection; by introducing artificial intelligence into log anomaly detection, the method not only can effectively help operation and maintenance personnel to find abnormal behaviors or abnormal conditions in large-scale log data, thereby improving the safety and robustness of network equipment, but also greatly reducing the workload of the operation and maintenance personnel; by preprocessing the log data before log abnormality detection, the method can ensure that too many errors and irrelevant data are not introduced in the whole process from data acquisition, storage to analysis and visualization, and meanwhile, the data can be arranged according to a certain sequence, so that a person can find out obvious characteristics or trends and find out clues for solving problems, and different types of data adopt different methods for detecting abnormality, thereby effectively improving the accuracy and reducing the false alarm rate, and solving the problems in the background technology.
In order to solve the technical problems, the invention is realized by the following technical scheme:
the invention relates to a network equipment security log event detection system, which mainly comprises:
log collection and processing, including:
(1) Log recording
(2) Log storage
(3) Log backup
Data preprocessing, including:
(1) Data auditing
(2) Data screening
(3) Data ordering
(4) Data cleansing
Feature extraction, comprising:
(1) Data statistics
(2) Data analysis
(3) Data modeling
Abnormality detection, including:
(1) A machine learning algorithm comprising: (1) clustering algorithm
(2) Classification algorithm
(2) A deep learning algorithm comprising: (1) supervised learning
(2) Semi-supervised learning
(3) Unsupervised learning
(3) Natural language processing technique exception event response, comprising:
(1) Event detection and analysis
(2) Event classification and priority assessment
(3) Event response planning
(4) Event isolation and recovery
(5) Evidence collection and analysis
(6) Event reporting and communication
(7) Event post-processing and improvement.
Further, for log records in log collection and processing, the log records are divided according to different contents and purposes, including:
(1) login log
(2) System log
(3) Access log
(4) And (5) security log.
Further, for log storage in log collection and processing, the following notes are included:
(1) data security
(2) Format specification
(3) And (5) protecting confidential information.
Further, the data auditing in the data preprocessing comprises the following steps:
(1) accuracy audit
(2) Suitability checking
(3) Timeliness audit
(4) And (5) consistency auditing.
Further, for data screening in data preprocessing, two aspects are mainly included:
(1) removing some data which do not meet the requirements or data with obvious errors;
(2) screening out data meeting a certain specific condition, and eliminating data not meeting the specific condition.
Further, for data cleaning in data preprocessing, the process includes:
(1) carrying out format conversion on the security data with different paths, different sources and different formats;
(2) filtering garbage;
(3) data deduplication;
(4) and (5) cleaning the formats.
Further, for data analysis in feature extraction, the following means are included:
(1) statistical analysis
(2) Visual analysis
(3) And (5) association analysis.
The network equipment security log event detection method is realized by adopting the network equipment security log event detection system and comprises the following steps:
s1, collecting log data in network equipment to a central server for unified processing, and preparing a disaster recovery backup strategy to ensure that the data cannot be lost due to server faults, natural disasters and the like;
s2, preprocessing the collected log data to ensure that too many errors and irrelevant data are not introduced in the whole process from data collection, storage to analysis and visualization;
s3, extracting useful specific information such as time, IP address, URL, abnormal behavior and other characteristics from the log file data by using pattern matching or keyword extraction technology according to the acquired data so as to select characteristics which are significant for abnormal detection, thereby reducing the complexity of calculation and improving the detection effect;
s4, modeling data obtained through feature extraction, and detecting weblog data by using an abnormal detection algorithm based on artificial intelligence so as to timely generate abnormal behaviors in a network;
s5, after an abnormal event is found, corresponding measures are quickly taken for coping and processing.
Compared with the prior art, the invention has the following beneficial effects:
(1) The invention develops a safety log detection system which can completely meet the requirement of a specific network, can be applied to different kinds of network equipment, and effectively solves the problem of the difference of the network equipment so as to meet the requirement of all users on log collection and detection;
(2) According to the invention, by introducing artificial intelligence into log anomaly detection, the operation and maintenance personnel can be effectively helped to find abnormal behaviors or abnormal conditions in large-scale log data, so that the safety and robustness of network equipment are improved, and the workload of the operation and maintenance personnel is greatly reduced;
(3) The method can be used for preprocessing the log data before log abnormality detection so as to ensure that too many errors and irrelevant data are not introduced in the whole process from data acquisition, storage to analysis and visualization, and meanwhile, the data can be arranged according to a certain sequence, so that a person can find out obvious characteristics or trends and find out clues for solving problems, and different types of data adopt different methods for detecting abnormality, thereby effectively improving the accuracy and reducing the false alarm rate.
Of course, it is not necessary for any one product to practice the invention to achieve all of the advantages set forth above at the same time.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method for detecting security log events of a network device according to the present invention;
fig. 2 is a block diagram of a network device security log event detection system according to the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1-2, a system for detecting security log events of network equipment according to the present invention mainly includes:
1. log collection and processing: collecting log data in the network equipment to a central server for unified processing, including:
(1) Log recording
(2) Log storage
(3) And (3) log backup: a disaster recovery backup strategy is prepared to ensure that data is not lost due to server faults, natural disasters and the like
2. Data preprocessing: preprocessing the collected log data to ensure that too many errors and irrelevant data are not introduced in the whole process from data collection, storage to analysis and visualization, wherein the method comprises the following steps:
(1) Data auditing
(2) Data screening: after the data passes the auditing, the errors found in the auditing process are corrected as much as possible, and when the found data errors cannot be corrected or some data do not meet the analysis processing requirements and cannot be compensated, the data are required to be screened;
the function of data screening in data preprocessing is very important, and the quality of the screened data directly influences the follow-up network security and the accuracy of situation understanding
(3) Data sorting: the data are arranged in a certain order so that the personnel find some obvious characteristics or trends and find clues for solving the problems, besides, the sorting is also helpful for checking and correcting the data, providing basis for reclassifying or grouping and the like, and in some cases, the sorting is one of the purposes of analysis
(4) Data cleaning: removing noise data and irrelevant data in a source data set, processing missing data and cleaning dirty data, removing blank data fields, and identifying or deleting outliers and solving inconsistency to clean data by filling in missing value smooth noise data, thereby ensuring the accuracy of subsequent detection results
3. Feature extraction: extracting useful specific information such as time, IP address, URL, abnormal behavior and other characteristics from the log file data according to the collected data by using pattern matching or keyword extraction technology so as to select characteristics which are significant for abnormal detection, thereby reducing the complexity of calculation and improving the detection effect, and comprising the following steps:
(1) Data statistics
(2) Data analysis
(3) Data modeling
4. Abnormality detection: modeling data obtained through feature extraction, detecting the weblog data by using an anomaly detection algorithm based on artificial intelligence so as to timely generate anomaly behaviors in a network, wherein the method comprises the following steps of:
(1) Machine learning algorithm: identifying instances inconsistent with normal behavior by learning patterns and rules in the log data, including:
(1) clustering algorithm: similar samples can be grouped into one-class, helping us find samples that differ from normal behavior
(2) Classification algorithm: unknown samples can be classified as normal or abnormal by learning known samples
(2) A deep learning algorithm; the method for carrying out complex pattern recognition and data analysis by simulating the working mode of the human brain neural network comprises the following steps:
(1) supervised learning
(2) Semi-supervised learning
(3) Unsupervised learning
(3) Natural language processing technology: the text data can be analyzed and understood to find abnormal behavior therein
5. Abnormal event response: after an abnormal event is found, corresponding measures are quickly taken for coping and processing, including:
(1) Event detection and analysis: network security events are discovered and identified in time by implementing security monitoring and log analysis, including monitoring network traffic and system logs using tools such as Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS), and collecting and analyzing security event data using security information and event management Systems (SIEM)
(2) Event classification and priority assessment: classifying and evaluating the detected security events, determining the severity and priority thereof, classifying the detected security events into different levels according to the types and influence degrees of the events so as to reasonably allocate resources and take corresponding measures
(3) Event response planning: formulating and implementing network security event response plans, defining tasks and responsibilities of each stage, including establishing emergency response team, defining roles and responsibilities of team members, formulating emergency contact lists, and ensuring that team members possess necessary skills and knowledge
(4) Event isolation and recovery: after the security event occurs, measures are timely taken to isolate the affected system and network so as to prevent further spreading of the event, and at the same time, system recovery and repair work are carried out, including bug repair, malicious code removal, data backup recovery and the like, so as to recover the normal operation of the system
(5) Evidence collection and analysis: detailed investigation and analysis of security events, collection of relevant evidence and logs, which help to determine the cause and scope of impact of the event, and means and purposes of identifying attackers, while also providing basis for subsequent legal liability and security improvements
(6) Event reporting and communication: timely reporting the condition and processing result of the security event to the related party, including the internal management layer, the partner and the related supervision mechanism, and simultaneously, effectively communicating and coordinating with the related party to jointly cope with and solve the security event
(7) Event post-processing and improvement: summarizing and evaluating the processing of security events, identifying existing problems and deficiencies, and taking corresponding improvements including strengthening security training and conscious education, updating security policies and measures, strengthening system and network monitoring and protection, etc., to increase the overall network security level.
Wherein, for log record, divide according to different content and usage, include:
(1) log-in: logging information of all users in the network system, including information such as user name, password, login time, login IP address and the like, wherein a login log can be used for preventing illegal access, tracing malicious attacks, detecting the use condition of system loopholes and the like;
(2) system log: recording log information of the running condition of the network system, knowing the running condition of the network system and the generation reason of system errors through the system log, and playing an important role in system maintenance and performance optimization;
(3) access log: the log information of the access condition of the network system to the external user is recorded, wherein the log information comprises accessed websites, access time, access IP addresses, access pages and the like, and operations such as anti-fraud or traceability can be performed according to analysis of the access log, so that the method has important security significance;
(4) security log: the log information of the security events of the network system is recorded, wherein the log information comprises security events such as intrusion events, virus attacks, malicious programs and the like, and the security log can help security personnel to discover, check and process the security events in time so as to protect the security of the network system.
Wherein for log storage, the following notes are included:
(1) data security: the weblog is important safety data, the safety of the weblog is enhanced when the weblog is stored, the data leakage and the tampering are required to be prevented in the process of storing the weblog, the access control is enhanced, and the safety of a data chain is ensured;
(2) format specification: the weblog should be stored in a common format, such as: CSV, XML, ON, etc. to ensure that it can be resolved by legal reading and analysis supporting a large number of tools;
(3) protection of confidential information: in the process of recording the weblog, care should be taken to protect the security of the confidential information, and the log information with the confidential mark should be recorded and stored in an encrypted manner to ensure the security of the confidential information.
Wherein, for data auditing, include:
(1) checking accuracy: checking the data from the aspects of authenticity and accuracy of the data, wherein the auditing focus is to check errors occurring in the investigation process;
(2) and (5) checking applicability: checking the degree of the problem interpreted by the data according to the purpose of the data, wherein the degree specifically comprises whether the data is matched with the selected theme, the definition of the target overall, and the like;
(3) checking timeliness: checking whether the data is transmitted according to the set time, if not, checking the reason of not transmitting in time;
(4) consistency auditing: checking whether the data are consistent in different storage spaces, and whether the data connotation is inconsistent, contradiction or incompatibility or the like.
For data screening, two aspects are mainly included:
(1) removing some data which do not meet the requirements or data with obvious errors;
(2) screening out data meeting a certain specific condition, and eliminating data not meeting the specific condition.
Wherein, for data cleaning, the process comprises:
(1) carrying out format conversion on the security data with different paths, different sources and different formats;
(2) filtering garbage;
(3) data deduplication;
(4) and (5) cleaning the formats.
Wherein for data analysis, the following means are included:
(1) statistical analysis: analyzing the log file by adopting a statistical method, for example, searching for the characteristics of abnormal events, error codes or malicious operations and the like with highest occurrence frequency;
(2) visual analysis: visualizing the log data, for example, using a histogram, a pie chart, a line chart and the like to display, and analyzing the trend and abnormal characteristics of the data;
(3) correlation analysis: through association and interaction analysis between different events, rules between the different events are found, and therefore association features between the events are extracted.
The network equipment security log event detection method is realized by adopting the network equipment security log event detection system and comprises the following steps:
s1, collecting log data in network equipment to a central server for unified processing, and preparing a disaster recovery backup strategy to ensure that the data cannot be lost due to server faults, natural disasters and the like;
s2, preprocessing the collected log data to ensure that too many errors and irrelevant data are not introduced in the whole process from data collection, storage to analysis and visualization;
s3, extracting useful specific information such as time, IP address, URL, abnormal behavior and other characteristics from the log file data by using pattern matching or keyword extraction technology according to the acquired data so as to select characteristics which are significant for abnormal detection, thereby reducing the complexity of calculation and improving the detection effect;
s4, modeling data obtained through feature extraction, and detecting weblog data by using an abnormal detection algorithm based on artificial intelligence so as to timely generate abnormal behaviors in a network;
s5, after an abnormal event is found, corresponding measures are quickly taken for coping and processing.
The preferred embodiments of the invention disclosed above are intended only to assist in the explanation of the invention. The preferred embodiments are not exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best understand and utilize the invention. The invention is limited only by the claims and the full scope and equivalents thereof.

Claims (8)

1. A network device security log event detection system, comprising: log collection and processing, including:
(1) Log recording
(2) Log storage
(3) Log backup
Data preprocessing, including:
(1) Data auditing
(2) Data screening
(3) Data ordering
(4) Data cleansing
Feature extraction, comprising:
(1) Data statistics
(2) Data analysis
(3) Data modeling
Abnormality detection, including:
(1) A machine learning algorithm comprising:
(1) clustering algorithm
(2) Classification algorithm
(2) A deep learning algorithm comprising:
(1) supervised learning
(2) Semi-supervised learning
(3) Unsupervised learning
(3) Natural language processing technique
An abnormal event response, comprising:
(1) Event detection and analysis
(2) Event classification and priority assessment
(3) Event response planning
(4) Event isolation and recovery
(5) Evidence collection and analysis
(6) Event reporting and communication
(7) Event post-processing and improvement.
2. The network device security log event detection system of claim 1, wherein for log records in log collection and processing, the partitioning is performed according to different content and purposes, comprising:
(1) login log
(2) System log
(3) Access log
(4) And (5) security log.
3. A network device security log event detection system according to claim 1 wherein for log storage in log collection and processing, the following notes are included:
(1) data security
(2) Format specification
(3) And (5) protecting confidential information.
4. The network device security log event detection system of claim 1, wherein for data auditing in data preprocessing, comprising:
(1) accuracy audit
(2) Suitability checking
(3) Timeliness audit
(4) And (5) consistency auditing.
5. The system for detecting events in a security log of a network device according to claim 1, wherein the data filtering in the data preprocessing mainly comprises two aspects:
(1) removing some data which do not meet the requirements or data with obvious errors;
(2) screening out data meeting a certain specific condition, and eliminating data not meeting the specific condition.
6. The network device security log event detection system of claim 1 wherein for data cleansing in data preprocessing, the process comprises:
(1) carrying out format conversion on the security data with different paths, different sources and different formats;
(2) filtering garbage;
(3) data deduplication;
(4) and (5) cleaning the formats.
7. A network device security log event detection system according to claim 1, wherein for data analysis in feature extraction, comprising:
(1) statistical analysis
(2) Visual analysis
(3) And (5) association analysis.
8. A method for detecting a security log event of a network device according to claim 1, implemented by a system for detecting a security log event of a network device according to any one of claims 1 to 7, comprising the steps of:
s1, collecting log data in network equipment to a central server for unified processing, and preparing a disaster recovery backup strategy to ensure that the data cannot be lost due to server faults, natural disasters and the like;
s2, preprocessing the collected log data to ensure that too many errors and irrelevant data are not introduced in the whole process from data collection, storage to analysis and visualization;
s3, extracting useful specific information such as time, IP address, URL, abnormal behavior and other characteristics from the log file data by using pattern matching or keyword extraction technology according to the acquired data so as to select characteristics which are significant for abnormal detection, thereby reducing the complexity of calculation and improving the detection effect;
s4, modeling data obtained through feature extraction, and detecting weblog data by using an abnormal detection algorithm based on artificial intelligence so as to timely generate abnormal behaviors in a network;
s5, after an abnormal event is found, corresponding measures are quickly taken for coping and processing.
CN202311189838.5A 2023-09-15 2023-09-15 System and method for detecting security log event of network equipment Pending CN117176441A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311189838.5A CN117176441A (en) 2023-09-15 2023-09-15 System and method for detecting security log event of network equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311189838.5A CN117176441A (en) 2023-09-15 2023-09-15 System and method for detecting security log event of network equipment

Publications (1)

Publication Number Publication Date
CN117176441A true CN117176441A (en) 2023-12-05

Family

ID=88931623

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311189838.5A Pending CN117176441A (en) 2023-09-15 2023-09-15 System and method for detecting security log event of network equipment

Country Status (1)

Country Link
CN (1) CN117176441A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117749535A (en) * 2024-02-21 2024-03-22 金数信息科技(苏州)有限公司 Network traffic abnormality detection method and device

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117749535A (en) * 2024-02-21 2024-03-22 金数信息科技(苏州)有限公司 Network traffic abnormality detection method and device
CN117749535B (en) * 2024-02-21 2024-05-07 金数信息科技(苏州)有限公司 Network traffic abnormality detection method and device

Similar Documents

Publication Publication Date Title
CN101803337B (en) Intrusion detection method and system
Tianfield Cyber security situational awareness
Spyridopoulos et al. Incident analysis & digital forensics in SCADA and industrial control systems
CN107222472A (en) A kind of user behavior method for detecting abnormality under Hadoop clusters
CN112417477A (en) Data security monitoring method, device, equipment and storage medium
CN110020687B (en) Abnormal behavior analysis method and device based on operator situation perception portrait
CN112114995A (en) Process-based terminal anomaly analysis method, device, equipment and storage medium
CN101609493A (en) A kind of database SQL infusion protecting method based on self study
CN109347808B (en) Safety analysis method based on user group behavior activity
CN107016298B (en) Webpage tampering monitoring method and device
CN105812200A (en) Abnormal behavior detection method and device
CN112560029A (en) Website content monitoring and automatic response protection method based on intelligent analysis technology
CN107733902A (en) A kind of monitoring method and device of target data diffusion process
CN117176441A (en) System and method for detecting security log event of network equipment
EP2747365A1 (en) Network security management
CN110262949A (en) Smart machine log processing system and method
CN115883236A (en) Power grid intelligent terminal cooperative attack monitoring system
CN117454376A (en) Industrial Internet data security detection response and tracing method and device
CN116861446A (en) Data security assessment method and system
CN110598397A (en) Deep learning-based Unix system user malicious operation detection method
CN117786748A (en) Digital analysis management system and method based on Internet
CN116074092B (en) Attack scene reconstruction system based on heterogram attention network
Salazar et al. Monitoring approaches for security and safety analysis: application to a load position system
CN115567241A (en) Multi-site network perception detection system
CN116614258A (en) Network danger prediction model of security situation awareness system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination