CN115408236A - Log data auditing system, method, equipment and medium - Google Patents
Log data auditing system, method, equipment and medium Download PDFInfo
- Publication number
- CN115408236A CN115408236A CN202211063609.4A CN202211063609A CN115408236A CN 115408236 A CN115408236 A CN 115408236A CN 202211063609 A CN202211063609 A CN 202211063609A CN 115408236 A CN115408236 A CN 115408236A
- Authority
- CN
- China
- Prior art keywords
- log data
- data
- log
- layer
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/32—Monitoring with visual or acoustical indication of the functioning of the machine
- G06F11/323—Visualisation of programs or trace data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a log data auditing system, method, equipment and medium. The system comprises: the data acquisition layer is used for acquiring the log data of each application system, performing standardization processing to obtain the log data to be classified and sending the log data to the data precipitation layer; the data precipitation layer is used for classifying the log data to be classified to obtain the log data to be processed and sending the log data to be processed to the data processing layer; the data processing layer determines log data to be converted corresponding to the target field segment from the log data to be processed, calls a corresponding conversion rule to convert the log data to be converted to obtain the log data to be analyzed, and sends the log data to be analyzed to the data analysis layer; the data analysis layer is used for determining to-be-processed log data associated with the to-be-analyzed log data, further performing track analysis and determining an analysis result as a target audit result; and the application display layer determines a target display graph based on the target audit result. And realizing centralized audit of log data of each application system.
Description
Technical Field
The invention relates to the technical field of log data auditing, in particular to a log data auditing system, method, equipment and medium.
Background
In order to strengthen the safety protection of enterprise information and guarantee the safety of enterprise information systems and data, a certain industry pays high attention to the work of information safety, and the log audit system is used for auditing the logs of an enterprise based on the log audit system, wherein the log audit system is a system for comprehensively collecting logs (including operation, alarm, operation, message, state and the like) generated by safety equipment, network equipment, a database, a server, an application system, a host and other equipment commonly used in an enterprise IT system and storing, monitoring, auditing, analyzing, alarming, responding and reporting the logs.
The existing auditing system, such as system auditing and application auditing, only performs log auditing aiming at a certain application system in an enterprise business support system, and has the defects of single auditing method and insufficient auditing strategy, thus causing low auditing work efficiency.
Disclosure of Invention
The invention provides a log data auditing system, method, equipment and medium, which are used for realizing centralized auditing of log data of all application systems on an enterprise business system.
According to an aspect of the present invention, there is provided a log data auditing system, including: the system comprises a data acquisition layer, a data precipitation layer, a data processing layer, a data analysis layer and an application display layer; wherein the content of the first and second substances,
the data acquisition layer is used for acquiring log data of each application system, carrying out standardized processing on the log data to obtain log data to be classified, and sending the log data to be classified to the data precipitation layer; wherein the log data of the application system comprises: host log data, database log data, application log data, weblog data, middleware log data, or security log data;
the data precipitation layer is used for receiving the log data to be classified, classifying the log data to be classified based on a preset classification rule to obtain the log data to be processed, and sending the log data to be processed to the data processing layer;
the data processing layer is used for receiving the log data to be processed, determining the log data to be converted corresponding to the target field from the log data to be processed, calling a conversion rule corresponding to the target field to convert the log data to be converted to obtain the log data to be analyzed, and sending the log data to be analyzed to the data analysis layer;
the data analysis layer is used for receiving the log data to be analyzed, determining to-be-processed log data associated with the log data to be analyzed, performing track analysis on the basis of the log data to be analyzed and the to-be-processed log data associated with the log data to be analyzed, determining an analysis result as a target audit result, and sending the target audit result to the application presentation layer;
and the application display layer is used for receiving the target audit result, determining a target display graph corresponding to the target audit result based on the target audit result, and displaying the target display graph.
According to another aspect of the present invention, there is provided a log data auditing method, which is applied to a log data auditing system, where the log data auditing system includes a data acquisition layer, a data precipitation layer, a data processing layer, a data analysis layer, and an application presentation layer, and the log data auditing method includes:
acquiring log data of each application system through a data acquisition layer, carrying out standardization processing on the log data to obtain log data to be classified, and sending the log data to be classified to a data precipitation layer; wherein the log data of the application system comprises: host log data, database log data, application log data, weblog data, middleware log data, or security log data;
receiving the log data to be classified through a data precipitation layer, classifying the log data to be classified based on a preset classification rule to obtain the log data to be processed, and sending the log data to be processed to a data processing layer;
receiving the log data to be processed through a data processing layer, determining the log data to be converted corresponding to a target field from the log data to be processed, calling a conversion rule corresponding to the target field to convert the log data to be converted to obtain the log data to be analyzed, and sending the log data to be analyzed to a data analysis layer;
receiving the log data to be analyzed through a data analysis layer, determining to-be-processed log data associated with the log data to be analyzed, performing track analysis based on the log data to be analyzed and the to-be-processed log data associated with the log data to be analyzed, determining an analysis result as a target audit result, and sending the target audit result to an application presentation layer;
and receiving the target auditing result through an application display layer, determining a target display graph corresponding to the target auditing result based on the target auditing result, and displaying the target display graph.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the log data auditing method of any embodiment of the present invention.
According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for causing a processor to implement the log data auditing method according to any one of the embodiments of the present invention when executed.
According to the technical scheme of the embodiment of the invention, log data of each application system are obtained through a data acquisition layer, the log data are subjected to standardization processing to obtain log data to be classified, and the log data to be classified are sent to a data precipitation layer; wherein the log data of the application system comprises: host log data, database log data, application log data, weblog data, middleware log data, or security log data; receiving the log data to be classified through a data precipitation layer, classifying the log data to be classified based on a preset classification rule to obtain the log data to be processed, and sending the log data to be processed to a data processing layer; receiving the log data to be processed through a data processing layer, determining the log data to be converted corresponding to a target field from the log data to be processed, calling a conversion rule corresponding to the target field to convert the log data to be converted to obtain the log data to be analyzed, and sending the log data to be analyzed to a data analysis layer; receiving the log data to be analyzed through a data analysis layer, determining to-be-processed log data associated with the log data to be analyzed, performing track analysis based on the log data to be analyzed and the to-be-processed log data associated with the log data to be analyzed, determining an analysis result as a target audit result, and sending the target audit result to an application presentation layer; the target audit result is received through the application presentation layer, the target presentation graph corresponding to the target audit result is determined based on the target audit result, and the target presentation graph is displayed, so that the problems that the existing audit mode can only audit log data for a certain application system, the audit strategy is single, and the centralized audit cannot be performed on data and heterogeneous data from different sources, so that the audit work efficiency is low are solved, the centralized audit on all application systems on the service support system is realized, the centralized audit on the data and the heterogeneous data from different sources can be realized, the audit efficiency is improved, and the audit strategy is increased.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a structural diagram of a log data auditing system according to an embodiment of the present invention;
fig. 2 is a structural diagram of a log data auditing system according to a second embodiment of the present invention;
fig. 3 is a log data collection model diagram of a log data auditing system according to a second embodiment of the present invention;
fig. 4 is a diagram of a log label processing model of a log data auditing system according to a second embodiment of the present invention;
FIG. 5 is a layout diagram of a distributed search of a log data auditing system according to a second embodiment of the present invention;
fig. 6 is a design diagram of dynamic field parsing of a log data auditing system according to a second embodiment of the present invention;
fig. 7 is a flowchart of a log data auditing method according to a third embodiment of the present invention;
fig. 8 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Before the technical solution is introduced, an application scenario is exemplarily described. The technical scheme can be applied to the situation that the centralized audit needs to be carried out on the log data of each application system on the service support system, the log data audit is mainly used for ensuring the data safety and normal operation of the service support system, and the service support system comprises a plurality of application systems, such as an application system A and an application system B. On the basis, different application systems can generate corresponding log data, the log data on each application system can be obtained, the obtained log data from different sources are processed and analyzed in a centralized mode, analysis results are obtained and displayed, and auditing of the service support system is achieved.
Example one
Fig. 1 is a structural diagram of a log data auditing system according to an embodiment of the present invention, where this embodiment is applicable to a case where centralized auditing needs to be performed on logs of each application system, and may be implemented by the log data auditing system, where the log data auditing system may be implemented in a hardware and/or software form, and the log data auditing system apparatus may be configured at a PC end or a mobile terminal.
As shown in fig. 1, the system includes: the system comprises a data acquisition layer, a data precipitation layer, a data processing layer, a data analysis layer and an application display layer.
The data acquisition layer is used for acquiring log data of each application system, standardizing the log data to obtain log data to be classified, and sending the log data to be classified to the data precipitation layer; wherein the log data of the application system comprises: host log data, database log data, application log data, weblog data, middleware log data, or security log data. The application system may be each application system on the service support system, for example, a customer management system or other application systems, an enterprise user may generate corresponding log data when using each application system, may access different application systems to the log data auditing system, and may also access a cloud platform for storing log data of each application system to the log data auditing system, so that a data acquisition layer of the application auditing system obtains the log data of each application system. Specifically, corresponding ports can be configured between the log data auditing system and each application system, so that log data transmission is realized.
It can be understood that, since the application systems accessed by the business support system are developed by different developers, the format of the log data generated by different application systems and the mapping rule are generally different. On the basis, the acquired log data needs to be subjected to standardization processing, wherein the standardization processing refers to unifying the log data of different application systems, the log data of different formats can be unified into a certain format, for example, all the log data are unified into a log format for storage, the processed log data are used as the log data to be classified, and the log data to be classified are sent to a data precipitation layer, so that the data precipitation layer further processes the log data to be classified.
In this embodiment, each application system includes a corresponding host, a database, an application, a network system, a middleware, and a security protection system, and accordingly, corresponding log data, that is, host log data, database log data, application log data, network log data, middleware log data, and security log data, may be generated.
The data precipitation layer is used for receiving the log data to be classified, classifying the log data to be classified based on a preset classification rule to obtain the log data to be processed, and sending the log data to be processed to the data processing layer. The preset classification rule can be understood as a data classification rule preset by a user and is a rule for classifying the log data to be classified, and the log data to be classified can be classified according to the preset classification rule after the data-precipitation layer receives the log data to be classified, so that the data processing layer can process the log data more efficiently.
The data processing layer is used for receiving the log data to be processed, determining the log data to be converted corresponding to the target field from the log data to be processed, calling the conversion rule corresponding to the target field to convert the log data to be converted so as to obtain the log data to be analyzed, and sending the log data to be analyzed to the data analysis layer. The target field refers to a preset field, if the log data to be processed corresponding to the target field exists in the log data to be processed, the partial log data to be processed can be extracted and used as the log data to be converted and conversion processing is carried out, the conversion rule can be a processing rule of the log data to be processed, different target fields correspond to different conversion rules, the conversion rule can be stored in the log data auditing system in advance, the log data to be converted is converted through the log conversion rule, and the obtained data is the log data to be analyzed. Further, the log data to be analyzed can be sent to a data analysis layer, and the log data to be analyzed can be analyzed.
The data analysis layer is used for receiving the log data to be analyzed, determining to-be-processed log data associated with the log data to be analyzed, performing track analysis on the basis of the log data to be analyzed and the to-be-processed log data associated with the log data to be analyzed, determining an analysis result as a target audit result, and sending the target audit result to the application presentation layer; the log data to be processed refers to log data associated with the log data to be analyzed, it can be understood that different log data correspond to different user operations, and the different operations are also associated, and when a user operates on the application system, corresponding log data can be generated, for example, after the user performs a first operation step, the corresponding log data can be obtained, and after the user performs a second operation step, the corresponding log data can be obtained. At this time, if the log data corresponding to a certain operation step is to-be-analyzed data, the log data corresponding to the operation step associated with the step may be regarded as to-be-processed data. In practical application, backtracking may be performed based on the log data to be analyzed to determine log data to be processed related to the log data to be analyzed, and further performing trajectory analysis based on the log data to be analyzed and the log data to be processed, where the trajectory analysis may be understood as performing analysis and determination on an operation trajectory corresponding to the log data, for example, an operation trajectory of a user on an application system may be determined based on the log data to be analyzed and the log data to be processed.
Illustratively, if a user deletes a certain item in an application system, corresponding log data is generated, and if it is determined that the log data is to-be-analyzed log data, the to-be-processed log data associated with the log data can be determined, and then trajectory analysis is performed according to the to-be-analyzed log data and the to-be-processed log data, so as to determine under what condition the user deletes the certain item, and what operation is performed after deletion, that is, an operation trajectory of the user can be determined, the operation trajectory is taken as a target audit result, and the target audit result is sent to an application presentation layer.
And the application display layer is used for receiving the target audit result, determining a target display graph corresponding to the target audit result based on the target audit result, and displaying the target display graph. The target display diagram can be a pie diagram, a line diagram or a bar diagram, the audit result can be displayed in the target display diagram mode, and the background user can know the operation of the front-end user on the application system more clearly based on the target display diagram to determine whether the user performs misoperation or not, or whether the user uses the application system according to the regulation or not, so that the normal operation of the application system is ensured.
On the basis of the technical scheme, the data acquisition layer is further configured to perform log verification processing, automatic mapping processing, log analysis processing or information completion processing on the log data after the log data of each application system is acquired, and determine the processed log data as the log data to be classified. In practical application, firewall logs, intrusion Detection System (IDS) logs, risk prompt logs, server host logs, security gateway logs, application system logs and database logs in each application system can be collected in a centralized manner and accessed into distributed cloud storage or a big data platform, and then a log data auditing system can acquire corresponding log data from the distributed cloud storage and the big data platform and perform log verification processing, automatic mapping processing, log analysis processing or information completion processing on the log data.
On the basis of the technical scheme, the data precipitation layer is further used for classifying the log data to be classified based on a preset field or an organization structure so as to obtain the log data to be processed. The preset field is a field name preset by a user, after receiving log data to be classified, all the log data to be classified can be classified according to the field, for example, the log data can be classified according to a field a and a field B, specifically, all the log data to be classified can be classified according to the field a and the field B, that is, the log data to be classified is divided into two different types, one type of log data corresponds to the field a, one type of log data corresponds to the field B, the log data can also be analyzed according to an organization structure in an application system, in the organization structure, a management user and a common user correspond to each other, the log data of the common user is one type, the log data of the management user is one type, and of course, the log data corresponding to the management user or the common user can be further subdivided and is determined according to actual conditions.
On the basis of the above technical solution, the data processing layer is further configured to: if the field corresponding to the log data to be processed is the same as the target field, determining the log data to be processed as log data to be converted; and calling a target conversion rule corresponding to the target field, and converting the log data to be converted based on the target conversion rule to obtain the log data to be analyzed.
For example, the target field may be some important fields representing the business system, the content of the field corresponding to the field is important data belonging to the business system, if a certain log data to be processed corresponds to the target field, the certain log data to be processed is taken as log data to be converted, and a corresponding conversion rule is determined to process the certain log data to be converted, for example, desensitization processing is performed on the important log data to be converted.
On the basis of the above technical solution, the data analysis layer is further configured to: determining a corresponding log track based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed; and analyzing based on the log track and a preset track, and determining an abnormal result to take the abnormal result as a target audit result. The log track may be a user operation track corresponding to the log, the preset track may be understood as a preset user operation track, and specifically, after the log track is determined, if the log track is inconsistent with the preset track, it is indicated that the user operation track is abnormal, the analyzed result may be used as an abnormal result, and the abnormal result may be used as a target audit result, so as to perform subsequent display or early warning.
On the basis of the above technical solution, the data analysis layer is further configured to: determining a corresponding log track based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed; determining operation items, operation steps and operation frequency corresponding to the log data to be divided based on the log track; and establishing an operation portrait according to the operation item, the operation step and the operation frequency, and determining the operation portrait as a target audit result so that the application display layer can display the target audit result. The operation items may be operation items performed by a user in an application system, such as login operation items, the operation steps and the operation frequency may be specific steps performed by the user and the operation times of the operation items, the operation representation is used for representing preference attributes of the user in the operation behavior of the application system, specifically, log data to be analyzed and log data to be processed may be analyzed by some log analysis techniques, so as to obtain user operation items corresponding to the log data, the operation steps and the operation frequency, further analyze the operation habits of the user, and generate a corresponding representation, for example, after analyzing the log data, it is determined that the corresponding operation items include login operation items and data maintenance items, and the operation frequency is multiple times, so as to generate the representation of the user, which represents that the user is a management user who generally performs data maintenance in the application system. After the operation image is established, the operation image can be displayed through the application display layer.
On the basis of the technical scheme, the log data auditing system further comprises a log searching module; the log searching module is used for splitting a log field to be searched to obtain at least two log fields to be used corresponding to the log field to be searched; and determining target log data corresponding to the log fields to be searched based on at least two log fields to be used and at least two distributed index libraries.
The log search module can be a module developed based on solr technology, full-text search of logs can be realized based on the module, a field of a log to be searched can be a field name corresponding to the log which a user wants to search, for example, the user wants to search an A log in a log data auditing system, the field name corresponding to the A log is the field of the log to be searched, the user can input the field of the log to be searched into the log search module, the log search module splits the field of the log to be searched, and can split the field into three new fields, the new fields are input into the distributed index library, the number of the distributed index libraries is multiple, and then the corresponding result is inquired from the distributed index library and is used as target log data corresponding to the field of the log to be searched. The method is fuzzified full-text retrieval and is mainly realized based on distributed search and dynamic field analysis, only one field is arranged in a distributed index library in a storage mode, the field can be temporarily decomposed for query in the query process, retrieval second-level return data with the performance reaching the million-level is obtained, and the problems of huge audit data quantity and low retrieval efficiency are solved.
According to the technical scheme of the embodiment of the invention, log data of each application system are obtained through a data acquisition layer, the log data are subjected to standardization processing to obtain log data to be classified, and the log data to be classified are sent to a data precipitation layer; wherein the log data of the application system comprises: host log data, database log data, application log data, weblog data, middleware log data, or security log data; receiving the log data to be classified through a data precipitation layer, classifying the log data to be classified based on a preset classification rule to obtain the log data to be processed, and sending the log data to be processed to a data processing layer; receiving the log data to be processed through a data processing layer, determining the log data to be converted corresponding to a target field from the log data to be processed, calling a conversion rule corresponding to the target field to convert the log data to be converted to obtain the log data to be analyzed, and sending the log data to be analyzed to a data analysis layer; receiving the log data to be analyzed through a data analysis layer, determining to-be-processed log data associated with the log data to be analyzed, performing track analysis based on the log data to be analyzed and the to-be-processed log data associated with the log data to be analyzed, determining an analysis result as a target audit result, and sending the target audit result to an application presentation layer; the target audit result is received through the application presentation layer, the target presentation graph corresponding to the target audit result is determined based on the target audit result, and the target presentation graph is displayed, so that the problems that the existing audit mode can only audit log data for a certain application system, the audit strategy is single, and the centralized audit cannot be performed on data and heterogeneous data from different sources, so that the audit work efficiency is low are solved, the centralized audit on all application systems on an enterprise information system is realized, the centralized audit on the data and the heterogeneous data from different sources can be realized, the audit efficiency is improved, and the audit strategy is increased.
Example two
Fig. 2 is a structural diagram of a log data auditing system provided in the second embodiment of the present invention, where this embodiment is a preferred embodiment of the foregoing embodiment, and a specific implementation manner of this embodiment may refer to technical solutions in this embodiment. The technical terms that are the same as or corresponding to the above-mentioned embodiments are not described in detail herein.
As shown in FIG. 2, the log data auditing system comprises an acquisition layer, a data precipitation layer and a data processing layer
The data analysis layer and the application display layer are as follows: the system is mainly responsible for log collection work of resources such as a host, a database, an application, a network and the like;
data precipitation layer: preprocessing the collected logs;
a data processing layer: processing the collected logs in a label and index mode;
and a data analysis layer: analyzing and processing the log in modes of risk modeling, trajectory analysis, abnormal perception and the like; application presentation layer: and displaying the audit analysis result by the user access portal.
Log centralized design
A centralized management system for unified acquisition, distributed processing and storage of heterogeneous data is constructed, conversion from independent acquisition of each system to a centralized large data architecture at present is realized, and a distributed, reliable and highly available massive log aggregation platform system is formed.
The platform not only supports various data senders customized in the system for collecting data; but also has the ability to customize various types of data recipients. Meanwhile, the method has the functions of log standardization (log verification, automatic mapping, analysis and information completion), data screening, log preprocessing and the like.
The detailed acquisition model is shown in fig. 3:
log tagging processing design
A flexibly defined thematic audit model is realized based on a data tagging method, the combination and superposition of log types and log tags are supported, an audit log decision tree mining analysis strategy is induced and formed, the method is applied to scenes such as operation figures of personnel at foreground and background, user behavior track analysis and the like, not only realizes the requirement of information security audit for supporting actual service scenes of the service, but also greatly improves the scene and efficiency of security audit coverage.
The log tag processing model is shown in fig. 4:
user behavior data modeling design
By comprehensively analyzing the characteristics of the system, the user operation behavior data model with leading industry is planned and realized, various source data are integrated, a human-based complete behavior link model is established, and the complete event expression capability of 'human', 'source terminal', 'access channel', 'access resource', 'what operation' is provided.
And centrally processing logs in the range of the 6 types of service support systems, namely a host type log, an application type log, a database type log, a middleware type log, a weblog and a safety log. An algorithm is constructed on the basis of the fact that the abnormal behaviors are sparse behaviors in all operation behaviors, a behavior flow distribution diagram based on all hdfs off-line logs is constructed in a space by using a data structure of a behavior relation network, and the abnormal flow behaviors with sparse flow directions among behavior units are analyzed, so that all abnormal behavior logs of a user for system operation are screened out.
Obfuscated full-text retrieval
Full-text retrieval of the security logs is realized based on the Solr technology, retrieval second-level return data with performance reaching million levels is obtained, and the problems of huge audit data volume and low retrieval efficiency are solved.
The core of fuzzified full-text retrieval is realized based on distributed search and dynamic field resolution, wherein the design of the distributed search is as shown in FIG. 5: the design of dynamic field parsing is shown in fig. 6:
according to the technical scheme of the embodiment of the invention, log data of each application system are obtained through a data acquisition layer, the log data are subjected to standardization processing to obtain log data to be classified, and the log data to be classified are sent to a data precipitation layer; wherein the log data of the application system comprises: host log data, database log data, application log data, weblog data, middleware log data, or security log data; receiving the log data to be classified through a data precipitation layer, classifying the log data to be classified based on a preset classification rule to obtain the log data to be processed, and sending the log data to be processed to a data processing layer; receiving the log data to be processed through a data processing layer, determining the log data to be converted corresponding to a target field from the log data to be processed, calling a conversion rule corresponding to the target field to convert the log data to be converted to obtain the log data to be analyzed, and sending the log data to be analyzed to a data analysis layer; receiving the log data to be analyzed through a data analysis layer, determining to-be-processed log data associated with the log data to be analyzed, performing track analysis based on the log data to be analyzed and the to-be-processed log data associated with the log data to be analyzed, determining an analysis result as a target audit result, and sending the target audit result to an application presentation layer; the target audit result is received through the application presentation layer, the target presentation graph corresponding to the target audit result is determined based on the target audit result, and the target presentation graph is displayed, so that the problems that the existing audit mode can only audit log data for a certain application system, the audit strategy is single, and the centralized audit cannot be performed on data and heterogeneous data from different sources, so that the audit work efficiency is low are solved, the centralized audit on all application systems on an enterprise information system is realized, the centralized audit on the data and the heterogeneous data from different sources can be realized, the audit efficiency is improved, and the audit strategy is increased.
EXAMPLE III
Fig. 7 is a flowchart of a log data auditing method according to a third embodiment of the present invention, where this embodiment is applicable to a case where centralized auditing needs to be performed on logs of each application system, and may be implemented by a log data auditing system, where the log data auditing method may be implemented in a hardware and/or software manner, and the log data auditing system apparatus may be configured at a PC end or a mobile terminal. As shown in fig. 7, the method includes:
s310, acquiring log data of each application system through the data acquisition layer, carrying out standardization processing on the log data to obtain log data to be classified, and sending the log data to be classified to the data precipitation layer.
The log data of the application system comprises: host log data, database log data, application log data, weblog data, middleware log data, or security log data. The application system may be each application system on the service support system, for example, a customer management system or other application systems, an enterprise user may generate corresponding log data when using each application system, may access different application systems to the log data auditing system, and may also access a cloud platform for storing log data of each application system to the log data auditing system, so that a data acquisition layer of the application auditing system obtains the log data of each application system.
Specifically, corresponding ports can be configured between the log data auditing system and each application system, so that log data transmission is realized. The log format is stored, the processed log data is used as the log data to be classified, and the log data to be classified is sent to a data precipitation layer, so that the data precipitation layer further processes the log data to be classified.
S320, receiving the log data to be classified through the data precipitation layer, classifying the log data to be classified based on preset classification rules to obtain the log data to be processed, and sending the log data to be processed to the data processing layer.
Specifically, the preset classification rule can be understood as a data classification rule preset by a user and is a rule for classifying the log data to be classified, and the log data to be classified can be classified according to the preset classification rule after the data-based precipitation layer receives the log data to be classified, so that the data processing layer can process the log data more efficiently.
S330, receiving the log data to be processed through the data processing layer, determining the log data to be converted corresponding to the target field from the log data to be processed, calling the conversion rule corresponding to the target field to convert the log data to be converted to obtain the log data to be analyzed, and sending the log data to be analyzed to the data analysis layer.
The target field refers to a preset field, the conversion rule may be a processing rule for log data to be processed, and different target fields correspond to different conversion rules.
Specifically, if it is detected that log data to be processed corresponding to the target field exists in the log data to be processed, the log data to be processed may be extracted, and the extracted log data may be used as log data to be converted and subjected to conversion processing, the conversion rule may be stored in the log data auditing system in advance, the log data to be converted is converted according to the log conversion rule, and the obtained data is the log data to be analyzed. Further, the log data to be analyzed can be sent to a data analysis layer, and the log data to be analyzed can be analyzed.
S340, receiving the log data to be analyzed through the data analysis layer, determining the log data to be processed associated with the log data to be analyzed, performing track analysis based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed, determining an analysis result as a target audit result, and sending the target audit result to the application presentation layer.
Wherein the log data to be processed refers to log data associated with the log data to be analyzed.
Specifically, different log data correspond to different user operations, and the different operations are also related, so that when a user operates on an application system, the corresponding log data can be generated, for example, after the user performs a first operation step, the corresponding log data can be obtained, and after the user performs a second operation step, the corresponding log data can be obtained. At this time, if the log data corresponding to a certain operation step is the data to be analyzed, the log data corresponding to the operation step associated with the step may be regarded as the data to be processed. In practical application, backtracking may be performed based on the log data to be analyzed to determine the log data to be processed related to the log data to be analyzed, and further performing trajectory analysis based on the log data to be analyzed and the log data to be processed, where the trajectory analysis may be understood as performing analysis and determination on an operation trajectory corresponding to the log data, for example, an operation trajectory of a user on an application system may be determined based on the log data to be analyzed and the log data to be processed.
And S350, receiving the target audit result through the application presentation layer, determining a target presentation graph corresponding to the target audit result based on the target audit result, and presenting the target presentation graph.
Wherein, the target display chart can be a pie chart, a line chart or a bar chart.
Specifically, the auditing result can be displayed in the mode of the target display diagram, and the background user can clearly know the operation of the front-end user on the application system based on the target display diagram to determine whether the user performs misoperation or whether the user uses the application system according to the rule, so that the normal operation of the application system is ensured.
According to the technical scheme of the embodiment of the invention, the log data of each application system is obtained through a data acquisition layer, the log data is subjected to standardization processing to obtain the log data to be classified, and the log data to be classified is sent to a data precipitation layer; wherein the log data of the application system comprises: host log data, database log data, application log data, weblog data, middleware log data, or security log data; receiving the log data to be classified through a data precipitation layer, classifying the log data to be classified based on a preset classification rule to obtain the log data to be processed, and sending the log data to be processed to a data processing layer; receiving the log data to be processed through a data processing layer, determining the log data to be converted corresponding to a target field from the log data to be processed, calling a conversion rule corresponding to the target field to convert the log data to be converted to obtain the log data to be analyzed, and sending the log data to be analyzed to a data analysis layer; receiving the log data to be analyzed through a data analysis layer, determining to-be-processed log data associated with the log data to be analyzed, performing track analysis based on the log data to be analyzed and the to-be-processed log data associated with the log data to be analyzed, determining an analysis result as a target audit result, and sending the target audit result to an application presentation layer; the target audit result is received through the application presentation layer, the target presentation graph corresponding to the target audit result is determined based on the target audit result, and the target presentation graph is displayed, so that the problems that the existing audit mode can only audit log data for a certain application system, the audit strategy is single, and the centralized audit cannot be performed on data and heterogeneous data from different sources, so that the audit work efficiency is low are solved, the centralized audit on all application systems on an enterprise information system is realized, the centralized audit on the data and the heterogeneous data from different sources can be realized, the audit efficiency is improved, and the audit strategy is increased.
Example four
Fig. 8 is a schematic structural diagram of an electronic device according to a fourth embodiment of the present invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 8, the electronic device 40 includes at least one processor 41, and a memory communicatively connected to the at least one processor 41, such as a Read Only Memory (ROM) 42, a Random Access Memory (RAM) 43, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 41 may perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 42 or the computer program loaded from a storage unit 48 into the Random Access Memory (RAM) 43. In the RAM 43, various programs and data necessary for the operation of the electronic apparatus 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other via a bus 44. An input/output (I/O) interface 45 is also connected to bus 44.
A plurality of components in the electronic device 40 are connected to the I/O interface 45, including: an input unit 46 such as a keyboard, a mouse, or the like; an output unit 47 such as various types of displays, speakers, and the like; a storage unit 48 such as a magnetic disk, optical disk, or the like; and a communication unit 49 such as a network card, modem, wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunication networks.
In some embodiments, the log data auditing method may be implemented as a computer program tangibly embodied in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 40 via the ROM 42 and/or the communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the log data auditing method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the log data auditing method by any other suitable means (e.g., by way of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user may provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.
The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A log data auditing system, comprising: the system comprises a data acquisition layer, a data precipitation layer, a data processing layer, a data analysis layer and an application display layer; wherein the content of the first and second substances,
the data acquisition layer is used for acquiring log data of each application system, standardizing the log data to obtain log data to be classified, and sending the log data to be classified to the data precipitation layer; wherein the log data of the application system comprises: host log data, database log data, application log data, weblog data, middleware log data, or security log data;
the data precipitation layer is used for receiving the log data to be classified, classifying the log data to be classified based on a preset classification rule to obtain the log data to be processed, and sending the log data to be processed to the data processing layer;
the data processing layer is used for receiving the log data to be processed, determining the log data to be converted corresponding to the target field from the log data to be processed, calling a conversion rule corresponding to the target field to convert the log data to be converted to obtain the log data to be analyzed, and sending the log data to be analyzed to the data analysis layer;
the data analysis layer is used for receiving the log data to be analyzed, determining to-be-processed log data associated with the log data to be analyzed, performing track analysis on the basis of the log data to be analyzed and the to-be-processed log data associated with the log data to be analyzed, determining an analysis result as a target audit result, and sending the target audit result to the application presentation layer;
and the application display layer is used for receiving the target audit result, determining a target display graph corresponding to the target audit result based on the target audit result, and displaying the target display graph.
2. The system of claim 1, wherein the data acquisition layer is further configured to:
after the log data of each application system is obtained, log verification processing, automatic mapping processing, log analysis processing or information completion processing are carried out on the log data, and the processed log data are determined as the log data to be classified.
3. The system of claim 1, wherein the data precipitation layer is further configured to:
and classifying the log data to be classified based on a preset field or an organization structure to obtain the log data to be processed.
4. The system of claim 1, wherein the data processing layer is further configured to:
if the field corresponding to the log data to be processed is the same as the target field, determining the log data to be processed as log data to be converted;
and calling a target conversion rule corresponding to the target field, and converting the log data to be converted based on the target conversion rule to obtain the log data to be analyzed.
5. The system of claim 1, wherein the data analysis layer is further configured to:
determining a corresponding log track based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed;
and analyzing based on the log track and a preset track, and determining an abnormal result to take the abnormal result as a target audit result.
6. The system of claim 1, wherein the data analysis layer is further configured to:
determining a corresponding log track based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed;
determining operation items, operation steps and operation frequency corresponding to the log data to be divided based on the log track;
and establishing an operation portrait according to the operation items, the operation steps and the operation frequency, and determining the operation portrait as a target audit result so that the application display layer can display the target audit result.
7. The system of claim 1, further comprising: a log searching module; wherein the content of the first and second substances,
the log searching module is used for splitting the log fields to be searched to obtain at least two log fields to be used corresponding to the log fields to be searched;
and determining target log data corresponding to the log fields to be searched based on at least two log fields to be used and at least two distributed index libraries.
8. The log data auditing method is applied to a log data auditing system, the log data auditing system comprises a data acquisition layer, a data precipitation layer, a data processing layer, a data analysis layer and an application presentation layer, and the log data auditing method comprises the following steps:
acquiring log data of each application system through a data acquisition layer, carrying out standardization processing on the log data to obtain log data to be classified, and sending the log data to be classified to a data precipitation layer; wherein the log data of the application system comprises: host log data, database log data, application log data, weblog data, middleware log data, or security log data;
receiving the log data to be classified through a data precipitation layer, classifying the log data to be classified based on a preset classification rule to obtain the log data to be processed, and sending the log data to be processed to a data processing layer;
receiving the log data to be processed through a data processing layer, determining the log data to be converted corresponding to a target field from the log data to be processed, calling a conversion rule corresponding to the target field to convert the log data to be converted to obtain the log data to be analyzed, and sending the log data to be analyzed to a data analysis layer;
receiving the log data to be analyzed through a data analysis layer, determining to-be-processed log data associated with the log data to be analyzed, performing track analysis based on the log data to be analyzed and the to-be-processed log data associated with the log data to be analyzed, determining an analysis result as a target audit result, and sending the target audit result to an application presentation layer;
and receiving the target auditing result through an application presentation layer, determining a target presentation graph corresponding to the target auditing result based on the target auditing result, and presenting the target presentation graph.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor;
wherein the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the log data auditing method of claim 8.
10. A computer readable storage medium having stored thereon computer instructions for causing a processor, when executed, to implement the log data auditing method of claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211063609.4A CN115408236A (en) | 2022-09-01 | 2022-09-01 | Log data auditing system, method, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211063609.4A CN115408236A (en) | 2022-09-01 | 2022-09-01 | Log data auditing system, method, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115408236A true CN115408236A (en) | 2022-11-29 |
Family
ID=84162957
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211063609.4A Pending CN115408236A (en) | 2022-09-01 | 2022-09-01 | Log data auditing system, method, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115408236A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116028461A (en) * | 2023-01-06 | 2023-04-28 | 北京志行正科技有限公司 | Log audit system based on big data |
-
2022
- 2022-09-01 CN CN202211063609.4A patent/CN115408236A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116028461A (en) * | 2023-01-06 | 2023-04-28 | 北京志行正科技有限公司 | Log audit system based on big data |
| CN116028461B (en) * | 2023-01-06 | 2023-09-19 | 北京志行正科技有限公司 | Log audit system based on big data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11586972B2 (en) | Tool-specific alerting rules based on abnormal and normal patterns obtained from history logs | |
| US7251584B1 (en) | Incremental detection and visualization of problem patterns and symptoms based monitored events | |
| US20170109657A1 (en) | Machine Learning-Based Model for Identifying Executions of a Business Process | |
| US20170109676A1 (en) | Generation of Candidate Sequences Using Links Between Nonconsecutively Performed Steps of a Business Process | |
| US20170109668A1 (en) | Model for Linking Between Nonconsecutively Performed Steps in a Business Process | |
| JP2018045403A (en) | Abnormality detection system and abnormality detection method | |
| US11042525B2 (en) | Extracting and labeling custom information from log messages | |
| CN109471783B (en) | Method and device for predicting task operation parameters | |
| US20170109639A1 (en) | General Model for Linking Between Nonconsecutively Performed Steps in Business Processes | |
| CN111414376A (en) | Data early warning method and device | |
| CN115033463B (en) | System exception type determining method, device, equipment and storage medium | |
| US20170109638A1 (en) | Ensemble-Based Identification of Executions of a Business Process | |
| CN115686910A (en) | Fault analysis method and device, electronic equipment and medium | |
| CN115033876A (en) | Log processing method, log processing device, computer device and storage medium | |
| US11568344B2 (en) | Systems and methods for automated pattern detection in service tickets | |
| CN115408236A (en) | Log data auditing system, method, equipment and medium | |
| US20170109640A1 (en) | Generation of Candidate Sequences Using Crowd-Based Seeds of Commonly-Performed Steps of a Business Process | |
| CN110879771A (en) | Log analysis system for user anomaly detection based on keyword sequence mining | |
| CN115048352B (en) | Log field extraction method, device, equipment and storage medium | |
| US20170109637A1 (en) | Crowd-Based Model for Identifying Nonconsecutive Executions of a Business Process | |
| CN116955856A (en) | Information display method, device, electronic equipment and storage medium | |
| CN116755974A (en) | Cloud computing platform operation and maintenance method and device, electronic equipment and storage medium | |
| CN116225848A (en) | Log monitoring method, device, equipment and medium | |
| CN114756301B (en) | Log processing method, device and system | |
| CN115794744A (en) | Log display method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |