CN115168297A - Bypassing log auditing method and device - Google Patents

Bypassing log auditing method and device Download PDF

Info

Publication number
CN115168297A
CN115168297A CN202110368338.2A CN202110368338A CN115168297A CN 115168297 A CN115168297 A CN 115168297A CN 202110368338 A CN202110368338 A CN 202110368338A CN 115168297 A CN115168297 A CN 115168297A
Authority
CN
China
Prior art keywords
log
detour
matching
module
storage
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110368338.2A
Other languages
Chinese (zh)
Inventor
饶品波
王鑫
马一骏
陈若鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Mobile Communications Group Co Ltd
China Mobile Group Jiangsu Co Ltd
Original Assignee
China Mobile Communications Group Co Ltd
China Mobile Group Jiangsu Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Mobile Communications Group Co Ltd, China Mobile Group Jiangsu Co Ltd filed Critical China Mobile Communications Group Co Ltd
Priority to CN202110368338.2A priority Critical patent/CN115168297A/en
Publication of CN115168297A publication Critical patent/CN115168297A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/1734Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/1805Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815Journaling file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention provides a detour log auditing method, a detour log auditing device, electronic equipment and a storage medium, wherein the method comprises the following steps: carrying out distributed storage on the collected log data according to a distributed system infrastructure; performing information completion on the distributed stored log data to obtain a standardized log to be put in storage; marking the bypassing type of the standardized logs to be put in storage based on a preset bypassing analysis processing rule; and dispatching the standardized logs to be put into storage corresponding to the different detour types to corresponding auditing task windows for auditing. According to the method, the log data are collected and stored through the distributed system infrastructure, and different types of bypassing behaviors are marked, so that the collection process of the log data is more efficient, and the obtained log audit result is more accurate.

Description

Bypassing log auditing method and device
Technical Field
The invention relates to the technical field of computer information system security, in particular to a detour log auditing method and device.
Background
In the 4A system, the bypassing behavior refers to all behaviors which do not go through the bastion machine login, and the bypassing 4A causes the problems that accurate recording cannot be carried out, the safety problem is difficult to trace and the like. According to the relevant regulations of the national information system safety, safety audit is definitely required to be carried out on more than two levels of information systems in the aspects of network safety, host safety, application safety and the like, and log audit is taken as a main audit means.
In the prior art, the bypassing behavior is audited, which mainly comprises the steps of compressing a large-scale log through classification compression, screening a class bypassing event from the log, and further screening the class bypassing event.
However, the prior art log auditing method for the detour behavior lacks efficiency and accuracy.
Disclosure of Invention
The invention provides a bypass log auditing method which is used for overcoming the defects of lack of efficiency and accuracy in the prior art of the bypass log auditing and realizing the recording and monitoring of the bypass.
The invention provides a detour log auditing method, which comprises the following steps:
carrying out distributed storage on the collected log data according to a distributed system infrastructure;
performing information completion on the distributed stored log data to obtain a standardized log to be put in storage;
marking the bypassing type of the standardized logs to be put in storage based on a preset bypassing analysis processing rule;
and dispatching the standardized logs to be put into storage corresponding to the different detour types to corresponding auditing task windows for auditing.
The invention also provides a detour log auditing device, which comprises:
the storage module is used for carrying out distributed storage on the acquired log data according to a distributed system infrastructure;
the completion module is used for performing information completion on the log data stored in a distributed mode to obtain a standardized log to be put into a warehouse;
the marking module is used for marking the bypassing type of the standardized logs to be put in storage based on a preset bypassing analysis processing rule;
and the scheduling module is used for scheduling the standardized logs to be put into storage corresponding to the different detour types to the corresponding auditing task window for auditing.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the steps of any one of the bypass log auditing methods.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the bypass log auditing method according to any of the above-described methods.
According to the bypassing log auditing method provided by the invention, the log data are acquired and stored through the distributed system infrastructure, and different types of bypassing behaviors are marked, so that the log data acquisition process is more efficient, and the obtained log auditing result is more accurate.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a bypass log auditing method provided by the present invention;
FIG. 2 is a general architectural diagram of a distributed storage graph provided by the present invention;
FIG. 3 is a schematic flow diagram of Hadoop log collection scheduling provided by the present invention;
FIG. 4 is a schematic flow diagram of a standardized service provided by the present invention;
FIG. 5 is a schematic flow diagram of a detour analysis service provided by the present invention;
FIG. 6 is a second flowchart illustrating a bypass log auditing method according to the present invention;
FIG. 7 is a schematic structural diagram of a bypass log auditing apparatus provided by the present invention;
FIG. 8 is a schematic structural diagram of an electronic device provided by the present invention;
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The detour log auditing method provided by the embodiment of the application can be applied to a 4A system, wherein the 4A system refers to a solution of a unified security management platform, and defines Authentication, account number accounting, authorization and Audit Audit, namely, the identity Authentication, authorization, accounting and Audit are defined as four major components of network security, and the status and the role of the identity Authentication in the whole network security system are established.
Fig. 1 is a schematic flow diagram of a bypass log auditing method provided in an embodiment of the present invention, and as shown in fig. 1, a bypass log auditing method provided in an embodiment of the present invention includes:
step 101, performing distributed storage on the collected log data according to a distributed system infrastructure.
Specifically, in the embodiment of the present invention, the log data may be divided into host login log information and database login log information, and different log information may be collected in different manners.
The host login log and the database login log in the embodiment of the invention may specifically include: login user name, user login IP address, login date and time, login times and the like. The log information can be written into a file or separated into different log files, and the corresponding files are uploaded to the log collection server in time for subsequent downloading and auditing. Different log information is collected in different ways.
For example, the user login record on the 4A system host is obtained in a Linux last command mode, the login information of the user can be displayed from back to front in reverse time order, the corresponding record is displayed according to the user name, the terminal or the time, and the method can be used for checking suspicious IP login.
As another example, through Structured Query Language, SQL Structured Query Language, is used to access and operate a database system. Creating SQL sentences, inquiring information such as operating system user names, programs being executed, login times, time and the like from the V $ session table of the Oracle dynamic performance table, and also being used for inquiring the authority of the user.
The Distributed System infrastructure Hadoop is used for realizing a Distributed File System, namely a Hadoop Distributed File System, which is called HDFS for short. HDFS is highly fault tolerant and provides high transmission rates to access data, storing data on all storage nodes in a Hadoop cluster. Hadoop is a software framework that can perform distributed processing on large amounts of data, and assumes that computing elements and storage will fail, thus maintaining multiple copies of working data to work, ensuring that processing can be redistributed to the failed nodes. Hadoop accelerates the data processing speed through parallel processing.
The dispatching of the log acquisition thread is carried out in a Hadoop mode, numerous log data information can be classified and stored, and the efficiency of log data acquisition is improved. Fig. 2 is a schematic diagram of the general architecture of a distributed storage map provided by the present invention.
As shown in fig. 2, the most core designs of the Hadoop framework are HDFS and MapReduce, where the HDFS provides storage for a large amount of log data, and the MapReduce provides computation for subsequent operations on the large amount of log data. Hbase and Hadoop Database are distributed storage systems which are high in reliability, high in performance, nematic and telescopic, and large-scale structured storage clusters can be built by utilizing the HBase technology. HBase provides a function of storing collected log data on top of Hadoop.
And 102, completing information of the log data stored in a distributed mode to obtain a standardized log to be put into a warehouse.
Specifically, after the collected log data is classified and stored by Hadoop, log data information is not complete, and the data itself and the collection process may be lost, so that some supplement or expansion needs to be performed on the content of the data. The embodiment of the invention completes the primary account information and the secondary account information.
For example, according to the matching log of the slave account name and the resource ID, if the matching is successful, the slave account information is supplemented, and the slave account ID, the slave account status, the slave account type, the slave account extension type, the slave account role and the like are included.
Complementing the primary account number, for example, when the primary account number is not empty, using primary account number name field matching, and complementing the information if the primary account number is successful; if the primary account is empty, matching is carried out according to the person responsible for the secondary account, and if the matching is successful, the information is completed; and if the information of the principal of the slave account is empty, searching the master-slave binding relationship, matching according to the primary account under the condition of binding only one primary account, and completing the information if the matching is successful.
And after completing the log data stored in the distributed mode, obtaining standardized log data information which is used as the basis of subsequent operation. And the log data information stored in a distributed manner is completed, so that the accuracy of the log audit result can be further improved.
103, marking the bypassing type of the standardized log to be put in storage based on a preset bypassing analysis processing rule.
Specifically, after obtaining the standardized log information to be put in storage, the data information needs to be audited. The auditing rule or the auditing algorithm is required to be preset, and the standardized logs are stored in the corresponding auditing queues according to the category distributed auditing algorithm in the embodiment of the invention for type marking. The collected detour logs can be divided into: personnel detour, program detour, emergency detour, internet detour, terminal detour, skip detour, suspected detour and the like.
The preset detour analysis processing rule is that various original logs are used as analyzed objects, matching baselines are set for various detour behaviors, and if matching is successful, the types of the detour behaviors are marked.
For example, the person detours by matching the slave account name, the source host, the source IP address field, the resource ID, the resource group ID, and the like in sequence, configuring the information such as the source IP, the person, the resource ID, and the like which cannot be logged in advance as a blacklist, matching the blacklist according to the slave account number and the login source address recorded in the log, and if the information can be matched, determining that the person detours.
For another example, the program bypasses, the slave account name, the source host, the source IP address segment, the resource ID and the resource group ID are sequentially matched, and if the white list is not matched, the program bypasses are marked; configuring information such as accessible source IP and resource ID into a white list in advance, and determining that the program bypasses if the information is matched according to the information accessed by the program and the information cannot be matched.
And 104, scheduling the standardized logs to be put into storage corresponding to the different detour types to corresponding auditing task windows for auditing.
Specifically, the type of the detour behavior is marked for the purpose of log audit in the following process. The bypassing of 4A causes problems that the behavior cannot be recorded as a reference, and the safety problem is difficult to trace. And marking different types of bypassing behaviors so as to facilitate query display of a bypassing log, bypassing related statistics and generation of a bypassing report.
And scheduling different types of bypassing behaviors to audit task windows of corresponding categories through a scheduler for auditing. The scheduler can be provided with a scheduling monitor for monitoring whether the detour log data received by each audit task window completes the audit, if so, the detour log data is scheduled to the cache database, and as the size of the cache database is continuously increased along with the increase of time, the detour log audit result stored in the cache database can be regularly cleaned.
According to the bypassing log auditing method, distributed acquisition and storage are carried out on log data, bypassing behaviors of different types are marked, the log acquisition process is more efficient, and the accuracy of the log auditing result of the bypassing behaviors is improved.
Optionally, before performing distributed storage on the collected log data according to a distributed system infrastructure, the method further includes:
collecting and configuring a log collection thread, wherein the log collection thread comprises resource type log attribute configuration and resource log attribute configuration; the resource type log attribute configuration refers to configuring resource logs of different types, and the resource log attribute configuration refers to configuring different acquisition modes for the resource logs of different types;
and collecting different types of log data through different collection modes.
Specifically, before the log data is stored in a distributed manner according to the distributed system infrastructure, the log data needs to be collected.
In the embodiment of the invention, different types of log data are collected in different modes. For example, using a last command mode to obtain information such as an account name, a login IP address and login time logged in a 4A system host; the database resource mainly queries the corresponding user login information in the V $ session table by executing the SQL statement.
Before collecting log data, collecting and configuring log collecting threads, including resource type log attribute configuration and resource log attribute configuration, wherein the resource type log attribute configuration refers to information such as command parameters, log regular expressions, log fields and the like for different types of resource configuration; the resource log attribute configuration mainly aims at different acquisition modes of different resource configuration, such as last acquisition, jdbc acquisition and the like. The jdbc collection refers to executing a Java API in an SQL statement, and the Java Application Programming Interface can provide uniform access for various relational databases.
In the embodiment of the invention, a Hadoop mode is adopted to schedule a log collection thread, FIG. 3 is a schematic flow diagram of Hadoop log collection and scheduling provided by the invention, as shown in FIG. 3, collected logs are classified in a one-time Hadoop task scheduling process to form a plurality of tasks, including task 1, task 2, … and task n, each task includes a collected task job list, including C1/Reduce1 and C1/Reduce 2.
The detour log auditing method provided by the embodiment of the application classifies detour log data, matches different acquisition modes for each type of detour log data, and performs distributed storage on the acquired detour log data, so that the log acquisition process is more efficient, and the accuracy of the auditing result of detour behavior is improved.
Optionally, the completing information of the log data stored in a distributed manner to obtain a standardized log to be put into storage includes:
determining the number of processing threads and the number of warehousing threads in the configuration information;
performing log matching on the distributed stored log data through a linear processor based on the processing thread number and the warehousing thread number;
and if the log matching is confirmed to be successful, completing the log data stored in the distributed mode to obtain the standardized log to be stored in the warehouse.
Specifically, after the collected log data is stored in a distributed manner, the log data information needs to be corrected and completed, so that the accuracy of the audit result is improved.
Fig. 4 is a schematic flow diagram of the standardized service provided by the present invention, and as shown in fig. 4, after the bypass standardized service is started, since it is not necessary to submit a return value, a factory class execute method is executed to obtain an execution class configuration in the configuration file.
According to the configuration file, the required caches, namely the primary account cache and the secondary account cache, are started, an execute method is executed through a circular execution type list, and the processing thread number and the warehousing thread number in the configuration information are obtained through a database login execution type and a host login execution type.
After the processing thread number and the warehousing thread number in the configuration information are obtained, a reading thread is started based on an original log to be standardized, a primary account processor and a secondary account processor are transmitted to a linear processor, a plurality of linear processing data and data writing threads are started, the standardized log to be warehoused is obtained, and the standardized service is ended.
In the embodiment of the invention, the log data stored in a distributed manner is corrected and supplemented, and the completion standardization of the primary account information and the secondary account information is realized.
And performing completion standardization on the slave account information, performing matching log according to the slave account information and the resource ID combination, and supplementing the slave account information if matching is successful, wherein the slave account information comprises a slave account ID, a slave account state, a slave account type, a slave account expansion type, a slave account role and the like.
And (2) performing completion standardization on the primary account information, wherein the first step is as follows: and if the primary account name is not null, matching by using the primary account name field, and completing the information if the primary account name is successful. The second step is that: and under the condition that the name of the primary account is null, matching according to the person responsible for the secondary account, and completing the information if the matching is successful. The third step: and when the person in charge of the slave account does not exist, searching the master-slave binding relationship. And when only one primary account is bound, matching according to the primary account, and completing the information if the matching is successful.
According to the bypassing log auditing method provided by the embodiment of the invention, the accuracy of the log auditing result of the bypassing behavior is further improved by matching and completing the log data information which is acquired and stored in a distributed manner.
Optionally, the marking different detour types according to a preset detour analysis processing rule based on the standardized logs to be put in storage includes:
creating a processor set according to the analysis class configuration in the analysis service configuration file;
performing bypass log matching based on the processor set;
marking a bypass type for the standardized logs to be put in storage, and if matching is successful, marking the bypass type; if the matching is unsuccessful, marking the type as a suspected bypass type to obtain a matching result;
and merging and writing the matching results into an analysis log.
Specifically, fig. 5 is a schematic flow diagram of a detour analysis service provided by the present invention, and as shown in fig. 5, after collected logs are standardized, the standardized logs are stored in an audit queue corresponding to distributed audit according to a category distributed audit algorithm, and according to the collected detour logs, the method can be divided into: personnel detour, program detour, emergency detour, internet detour, terminal detour, jump detour, suspected detour and the like.
The 4A management platform provides related configuration functions of black and white list management, emergency IP management, internet IP pool management, terminal IP management, bypassing reservation management and the like.
Black and white list management: the system consists of a slave account name, a source host, a source IP address segment, a destination resource group and a destination resource, and is divided into a black list and a white list.
Emergency IP pool management: the source host, IP address and IP address section form an emergency IP configuration record.
And (3) managing an internet IP pool: the source host, IP address and IP address section form an Internet IP pool configuration record.
And (3) terminal IP management: and the primary account name and the IP address form a terminal IP configuration record.
And (3) detour reservation management: one detour reservation application comprises information such as a detour login account number, detour resources, detour resource types, an approver, a source IP, detour starting time, detour failure time, detour reasons and the like, and has the functions of applying, auditing, conducting detour reservation supplement and the like.
Properties, the names and the sequence of host resources and database resource related analysis processors are configured in a bypass analysis service configuration file, when the bypass analysis service is executed, bypass analysis is executed according to the configuration sequence, and the specific type of a bypass log is processed as follows:
people detour: matching the slave account name, the source host, the source IP address field, the resource ID and the resource group ID in sequence, and marking the matched blacklist as a person to bypass if the matched blacklist is successful; configuring information such as source IP, personnel and resource ID which cannot be logged into a blacklist in advance, matching the blacklist according to the secondary account number and the login source address recorded by the log, and if the blacklist can be matched, determining that the personnel detour.
And (3) program bypassing: matching the slave account name, the source host, the source IP address field, the resource ID and the resource group ID in sequence, and marking the program to bypass if the white list is not matched; configuring accessible information such as source IP and resource ID into a white list in advance, and regarding the information as program bypass if the information is matched according to the program access information and the information cannot be matched.
Emergency detouring: sequentially matching a source host, a source IP address and a source IP address section, and marking as emergency detour if matching is successful; and setting the source host and the source IP as emergency addresses in advance, and if the related information in the log can be matched, bypassing the emergency.
Bypassing the Internet: sequentially matching the source host, the IP address and the IP address section, and marking the source host, the IP address and the IP address section as an internet bypass if the matching is successful; and setting information such as an internet source host, an IP address and the like, and if the information is matched, determining that the Internet bypasses.
The terminal detours: matching the source IP address, and marking the source IP address as a terminal to bypass if the source IP address is successfully matched; and according to the IP address of the login terminal, if the IP address can be matched with the previously preset IP address, the terminal is considered to be a bypass.
Skipping and bypassing: if the source resource ID in the log is empty, according to the source IP address in the log, if the corresponding 4A resource can be obtained, the log is a skip self-4A resource log, and the log is marked as skip bypass; and acquiring a source address in the log record, and if the source address can match with the resource data in the 4A, regarding the source address as a skip bypass from other equipment.
Suspected detour: and if all the detour analysis scenes are not matched, marking the detour log as a suspected detour type log.
And matching the standardized detour log data according to the preset rule. Fig. 6 is a second schematic flow chart of the detour log auditing method provided by the present invention, and as shown in fig. 6, after the standardized detour log database is obtained, according to the preset detour analysis processing rule, the auditing engine analyzes the detour log category and the corresponding auditing task window according to the data entered therein.
In some embodiments, a listener may be provided in the detour log database, and the audit engine receives the detour log database listening data in real time. And storing the standardized log files in a bypassing log database by categories, such as storing a first category database in a database category 1, storing a second category database in a database category 2, and the like.
The monitor can be configured to a time period reminding mode or a data storage full reminding mode, the two modes are set according to actual needs, the time period reminding mode is used for databases of some important types, and the data storage full reminding mode is used for databases of some less important types.
And when the time period reminding mode condition or the data storage full reminding mode condition is reached, starting an auditing engine and calling a preset detour analysis processing rule. And the auditing engine analyzes the type of the detour log and a corresponding auditing task window according to a preset detour analysis processing rule and the received log data.
The scheduler schedules the log analysis result or the matching result output by the audit engine to a corresponding audit task window for auditing, a scheduling monitor can be arranged in the scheduler to monitor whether the detour log data received by each audit task window is audited, if the detour log data is audited, the detour log data is scheduled to the cache database, and the detour log audit result stored in the cache database is cleared regularly.
And marking different types of bypassing behaviors through auditing the bypassing log so as to facilitate query display of the bypassing log, statistics on the bypassing behaviors and the like. And the real-time monitoring of the audit engine and the scheduler is provided, the real-time scheduling audit of the bypassing log data is realized, the audit result is monitored in real time, and the storage efficiency of the cache database is improved.
According to the bypassing log auditing method provided by the embodiment of the invention, different types of bypassing behaviors are marked, so that the query and display of the bypassing log, the bypassing related statistics and the generation of a bypassing report are facilitated, and the problem of inaccurate log auditing of a 4A system caused by passive collection, popularization and the like of the log is solved.
Fig. 7 is a schematic structural diagram of a detour log auditing apparatus according to an embodiment of the present invention, and as shown in fig. 7, the detour log auditing apparatus according to the embodiment of the present invention includes a storage module 701, a completion module 702, a marking module 703, and a scheduling module 704, where:
the storage module 701 is used for performing distributed storage on the acquired log data according to a distributed system infrastructure;
a completion module 702, configured to perform information completion on the distributively stored log data to obtain a standardized log to be put into a warehouse;
a marking module 703, configured to mark a detour type for the standardized log to be warehoused based on a preset detour analysis processing rule;
and the scheduling module 704 is configured to schedule the standardized logs to be warehoused corresponding to the different detour types to corresponding audit task windows for auditing.
Optionally, the detour log auditing device further comprises a configuration module and an acquisition module;
the configuration module is used for carrying out acquisition configuration on the log acquisition thread, and comprises resource type log attribute configuration and resource log attribute configuration; the resource type log attribute configuration refers to configuring resource logs of different types, and the resource log attribute configuration refers to configuring different acquisition modes for the resource logs of different types;
the acquisition module is used for acquiring different types of log data in different acquisition modes.
Optionally, the completion module includes a determination module, a matching module, and a completion submodule;
the determining module is used for determining the number of processing threads and the number of warehousing threads in the configuration information;
the matching module is used for performing log matching on the distributed stored log data through a linear processor based on the processing thread number and the warehousing thread number;
and the completion submodule is used for completing the log data stored in the distributed mode to obtain the standardized logs to be put into storage if the log matching is successful.
Optionally, the marking module includes a creating module, a matching sub-module, and a merging module;
the creating module is used for creating a processor set according to the analysis class configuration in the analysis service configuration file;
the matching submodule is used for marking a bypass type on the standardized logs to be put in storage, and if the matching is successful, the bypass type is marked; if the matching is unsuccessful, marking the type as a suspected bypass type to obtain a matching result;
and the merging module is used for merging the matching results and writing the matching results into an analysis log.
Specifically, the detour log auditing device provided in the embodiment of the present invention can implement all method steps in the above method embodiment, and can achieve the same technical effects, and details of the same parts and beneficial effects as those of the method embodiment in this embodiment are not repeated herein.
Fig. 8 illustrates a physical structure diagram of an electronic device, and as shown in fig. 8, the electronic device may include: a processor (processor) 801, a communication Interface (Communications Interface) 802, a memory (memory) 803 and a communication bus 804, wherein the processor 801, the communication Interface 802 and the memory 803 complete communication with each other through the communication bus 804. The processor 801 may call logic instructions in the memory 803 to perform a bypass log audit method comprising:
carrying out distributed storage on the collected log data according to a distributed system infrastructure;
performing information completion on the distributed stored log data to obtain a standardized log to be put in storage;
marking the bypassing type of the standardized logs to be put in storage based on a preset bypassing analysis processing rule;
and dispatching the standardized logs to be put into storage corresponding to the different detour types to corresponding auditing task windows for auditing.
In addition, the logic instructions in the memory 803 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the detour log auditing method provided by the above methods, the method comprising:
carrying out distributed storage on the collected log data according to a distributed system infrastructure;
performing information completion on the distributed stored log data to obtain a standardized log to be put in storage;
marking the bypassing type of the standardized logs to be put in storage based on a preset bypassing analysis processing rule;
and dispatching the standardized logs to be put into storage corresponding to the different detour types to corresponding auditing task windows for auditing.
In yet another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program that when executed by a processor is implemented to perform the detour log auditing methods provided above, the method comprising:
carrying out distributed storage on the collected log data according to a distributed system infrastructure;
performing information completion on the distributed stored log data to obtain a standardized log to be put in storage;
marking the bypassing type of the standardized logs to be put in storage based on a preset bypassing analysis processing rule;
and dispatching the standardized logs to be put into storage corresponding to the different detour types to corresponding auditing task windows for auditing.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (10)

1. A bypass log auditing method is characterized by comprising the following steps:
carrying out distributed storage on the collected log data according to a distributed system infrastructure;
performing information completion on the distributed stored log data to obtain a standardized log to be put in storage;
marking the bypassing type of the standardized logs to be put in storage based on a preset bypassing analysis processing rule;
and dispatching the standardized logs to be put into storage corresponding to the different detour types to corresponding auditing task windows for auditing.
2. The method for auditing detour logs according to claim 1, wherein before performing distributed storage on the collected log data according to a distributed system infrastructure, the method further comprises:
collecting and configuring a log collection thread, wherein the log collection thread comprises resource type log attribute configuration and resource log attribute configuration; the resource type log attribute configuration refers to configuring resource logs of different types, and the resource log attribute configuration refers to configuring different acquisition modes for the resource logs of different types;
and collecting different types of log data through different collection modes.
3. The detour log auditing method according to claim 2, wherein the information completion of the log data stored in a distributed manner to obtain a standardized log to be put in storage comprises:
determining the number of processing threads and the number of warehousing threads in the configuration information;
performing log matching on the distributed stored log data through a linear processor based on the processing thread number and the warehousing thread number;
and if the log matching is confirmed to be successful, completing the log data stored in the distributed mode to obtain the standardized log to be stored in the warehouse.
4. The detour log auditing method according to claim 1, wherein the marking of different detour types according to a preset detour analysis processing rule based on the standardized to-be-warehoused log comprises:
creating a processor set according to the analysis class configuration in the analysis service configuration file;
performing bypass log matching based on the processor set;
marking a bypass type for the standardized logs to be put in storage, and if matching is successful, marking the bypass type; if the matching is unsuccessful, marking the type as a suspected bypass type to obtain a matching result;
and merging and writing the matching results into an analysis log.
5. A detour log audit device, comprising:
the storage module is used for carrying out distributed storage on the acquired log data according to a distributed system infrastructure;
the completion module is used for performing information completion on the log data stored in a distributed mode to obtain a standardized log to be put into a warehouse;
the marking module is used for marking the bypassing type of the standardized logs to be put in storage based on a preset bypassing analysis processing rule;
and the scheduling module is used for scheduling the standardized logs to be put into the warehouse corresponding to the different detour types to corresponding audit task windows for auditing.
6. The detour log audit device according to claim 5, further comprising a configuration module and an acquisition module;
the configuration module is used for carrying out acquisition configuration on the log acquisition thread, and comprises resource type log attribute configuration and resource log attribute configuration; the resource type log attribute configuration refers to configuring resource logs of different types, and the resource log attribute configuration refers to configuring different acquisition modes for the resource logs of different types;
the acquisition module is used for acquiring different types of log data in different acquisition modes.
7. The detour log audit device of claim 5 wherein the completion module includes a determination module, a matching module and a completion sub-module;
the determining module is used for determining the number of processing threads and the number of warehousing threads in the configuration information;
the matching module is used for performing log matching on the distributed stored log data through a linear processor based on the processing thread number and the warehousing thread number;
and the completion submodule is used for completing the log data stored in the distributed mode to obtain the standardized logs to be put into storage if the log matching is successful.
8. The detour log audit device of claim 5 wherein the marking module includes a creation module, a matching sub-module and a merging module;
the creating module is used for creating a processor set according to the analysis class configuration in the analysis service configuration file;
the matching submodule is used for marking a bypass type on the standardized logs to be put in storage, and if the matching is successful, the bypass type is marked; if the matching is unsuccessful, marking the type as a suspected bypass type to obtain a matching result;
and the merging module is used for merging the matching results and writing the matching results into an analysis log.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program implements the steps of the bypass log auditing method according to any one of claims 1 to 4.
10. A non-transitory computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, performs the steps of the detour log auditing method according to any one of claims 1 to 4.
CN202110368338.2A 2021-04-06 2021-04-06 Bypassing log auditing method and device Pending CN115168297A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110368338.2A CN115168297A (en) 2021-04-06 2021-04-06 Bypassing log auditing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110368338.2A CN115168297A (en) 2021-04-06 2021-04-06 Bypassing log auditing method and device

Publications (1)

Publication Number Publication Date
CN115168297A true CN115168297A (en) 2022-10-11

Family

ID=83476260

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110368338.2A Pending CN115168297A (en) 2021-04-06 2021-04-06 Bypassing log auditing method and device

Country Status (1)

Country Link
CN (1) CN115168297A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116028461A (en) * 2023-01-06 2023-04-28 北京志行正科技有限公司 Log audit system based on big data

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116028461A (en) * 2023-01-06 2023-04-28 北京志行正科技有限公司 Log audit system based on big data
CN116028461B (en) * 2023-01-06 2023-09-19 北京志行正科技有限公司 Log audit system based on big data

Similar Documents

Publication Publication Date Title
CN105631026A (en) Security data analysis system
CN109753596B (en) Information source management and configuration method and system for large-scale network data acquisition
CN111400288A (en) Data quality inspection method and system
CN110851324B (en) Log-based routing inspection processing method and device, electronic equipment and storage medium
US20130036127A1 (en) Document registry system
CN111611276A (en) Data query method, device and storage medium
CN109819019B (en) Monitoring and statistical analysis method and system for large-scale network data acquisition
CN112559525B (en) Data checking system, method, device and server
CN113836237A (en) Method and device for auditing data operation of database
CN115168297A (en) Bypassing log auditing method and device
CN111639016A (en) Big data log analysis method and device and computer storage medium
CN115567563B (en) Comprehensive transportation hub monitoring and early warning system based on end edge cloud and control method thereof
CN114416601B (en) Network security information acquisition engine and task management system and method
CN104317820B (en) Statistical method and device for report forms
CN116483903A (en) All-link data blood-edge relation identification method for multi-source heterogeneous data source
CN113220530B (en) Data quality monitoring method and platform
CN112965793B (en) Identification analysis data-oriented data warehouse task scheduling method and system
CN117131059A (en) Report data processing method, device, equipment and storage medium
CN113612832A (en) Streaming data distribution method and system
US10936571B1 (en) Undo based logical rewind in a multi-tenant system
CN115348185B (en) Control method and control device of distributed query engine
CN116170321B (en) Data collection method, device, equipment and storage medium for link tracking
CN116910352A (en) Report recommending method, device, equipment and medium based on artificial intelligence
CN116629816A (en) Human resource management and decision-making aid system and method based on big data, electronic equipment and storage medium
CN117834458A (en) Account string detection method and device, storage medium and electronic device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination