CN107818150A - A kind of log audit method and device - Google Patents
A kind of log audit method and device Download PDFInfo
- Publication number
- CN107818150A CN107818150A CN201710994900.6A CN201710994900A CN107818150A CN 107818150 A CN107818150 A CN 107818150A CN 201710994900 A CN201710994900 A CN 201710994900A CN 107818150 A CN107818150 A CN 107818150A
- Authority
- CN
- China
- Prior art keywords
- log
- audit
- daily record
- big data
- data platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/90335—Query processing
- G06F16/90344—Query processing by using string matching techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/552—Detecting local intrusion or implementing counter-measures involving long-term monitoring or reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/57—Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
- G06F21/577—Assessing vulnerabilities and evaluating computer system security
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F2221/03—Indexing scheme relating to G06F21/50, monitoring users, programs or devices to maintain the integrity of platforms
- G06F2221/034—Test or assess a computer or a system
Abstract
The invention discloses a kind of log audit method and device, specifically, by being parsed to the original log collected from major data platform component, field standardization maps and the action type of daily record and the thin item division processing of operation, it is possible to achieve the initial log of each component is standardized in the big data platform that source, form differ；Then, according to the audit requirement of big data security management and control, using corresponding audit regulation and analysis strategy, the daily record after standardization to each component in big data platform carries out automation audit analysis, to determine whether the management of big data platform and component and data access operation meet technical specification of security and management requirement.Compared with manual audit's mode, log audit method provided by the invention, audited by the standardization to big data platform assembly daily record with concentrating, operation that can be to big data platform assembly is carried out comprehensively and timely audited, and quickly finds potential safety hazard, positioning security problem.
Description
Technical field
The present invention relates to field of information security technology, more particularly to a kind of log audit method and device.
Background technology
With the continuous improvement of social informatization technology and the quick popularization of Internet technology, it is necessary to the data of processing
It is more and more, and big data platform has unrivaled superiority in the extensive storage of data and high-performance computing sector, energy
Efficient big data storage, calculating, O＆M and monitoring service are enough provided.However, the safety prevention measure of current big data platform
Lack standard and requirement, do not catch up with the development of big data platform own service demand, exist with data set, the height of data sharing
The unmatched defect of value business.Therefore, data safety risk and solution in big data platform itself and platform are explored, it is deep
Change big data security management and control scope and application field, be the emphasis studied at present.
For big data platform core component, such as HDFS (Hadoop Distributed File System, distributed text
Part system), Hive, HBase, YARN＆MR etc., the characteristics of its substantial amounts of information can all be stored in Operation Log file.Specifically,
Its Operation Log can be divided into two classes again, including component safeguards daily record and data access log, and the former have recorded platform management behaviour
Make such as point spread removal, node start and stop, Component service start and stop, the latter and have recorded user activity information and user operation instruction
Information.Therefore, the Operation Log of big data platform can be used for orientation problem reason and division accident responsibility in security incident.Phase
Answer, the concentration log audit of big data platform, study record, storage, collection, the mark of each component Operation Log of big data platform
Standardization processing and audit and alarm, promote audit strategy landed under big data environment be big data platform safety management and control weight
Want link.
At present, audited for the Operation Log of big data platform, generally by the safety manager of enterprise periodically from service
Original log information is checked on the node of component, or, partial log is checked by some big data management platforms, with artificial side
Formula is investigated and audit, to determine whether the management of platform and component and data access operation meet technical specification of security and management and want
Ask.
However, the characteristics of scale for being limited by big data platform is big, component and nodes are numerous, by manual type to big
The Operation Log of data platform is audited, and has that daily record is scattered, the scale of construction is big, and manual type wastes time and energy, efficiency and accuracy rate
The shortcomings that low.In addition, manual type requires very high to the professional degree of auditor, safety service is not only understood, it is also necessary to
Solve big data platform, including the infrastructure environment of platform and each component of platform and the management operating mechanism of service.
The content of the invention
The invention provides a kind of log audit method and device, to realize that the operation to big data platform assembly is carried out entirely
Face and timely audit, quickly find potential safety hazard, positioning security problem.
First aspect according to embodiments of the present invention, there is provided a kind of log audit method, this method include：
The original log collected from big data platform assembly is parsed, obtains effective day in the original log
Will field and the property value effectively corresponding to log field；
According to preset field mapping ruler, institute is carried out to property value corresponding to effective log field in the original log
Category standardization field is sorted out, and obtains initial log；
The keyword in effective log field in the initial log, carry out the action type of the initial log
Divided with thin item is operated, obtain standardizing daily record；
According to default audit regulation and analysis strategy, the standardization daily record to the big data platform assembly is audited.
Alternatively, according to default audit regulation and analysis strategy, the standardization daily record to the big data platform assembly is entered
Row audit, including：
Action type and the thin item of operation in the standardization daily record of the big data platform assembly, select corresponding day
Will content auditing point and audit regulation；
According to log content audit point and audit regulation, the log content of the standardization daily record is examined
Meter.
Alternatively, before being parsed to the original log collected from big data platform assembly, methods described also includes：
Saving options for the big data platform assembly daily record is opened, and the big data platform assembly daily record is set
Option of operation is preserved, wherein, the preservation option of operation includes the title of journal file, the store path of journal file, daily record
The size of file and the quantity of journal file.
Alternatively, the choosing method of the store path of the journal file includes：
Choose store path of the node of the big data platform assembly interior joint minimum number as journal file.
Alternatively, the acquisition mode of the original log of the big data platform assembly includes：
By syslog agreements, the original log that the big data platform assembly is sent is obtained；
Or
By way of FTP/SFTP, the original log of its preservation is gathered from the big data platform assembly.
Alternatively, after the standardization daily record to the big data platform assembly is audited, methods described also includes：
When log content in the standardization daily record be present and do not meet default audit regulation, then the corresponding safety of generation is pre-
Alert information.
Second aspect according to embodiments of the present invention, additionally provides a kind of log audit device, and the device includes：
Original log parsing module：For being parsed to the original log collected from big data platform assembly, obtain
Effective log field and the property value effectively corresponding to log field in the original log；
Standardize field classifying module：For according to preset field mapping ruler, to effective day in the original log
Property value corresponding to will field carries out said standard field and sorted out, and obtains initial log；
Action type division module：For the keyword in effective log field in the initial log, carry out
The action type of the initial log and the thin item division of operation, obtain standardizing daily record；
Standardize log audit module：For the default audit regulation of basis and analysis strategy, to the big data platform group
The standardization daily record of part is audited.
Alternatively, the standardization log audit module includes：
Audit strategy chooses submodule：For the action type in the standardization daily record according to the big data platform assembly
With the thin item of operation, corresponding log content audit point and audit regulation are selected；
Log content audit submodule：For according to the log content audit point and audit regulation, to the standard
The log content for changing daily record is audited.
Alternatively, described device also includes：
Logging option setup module：For opening saving options for the big data platform assembly daily record, and institute is set
The preservation option of operation of big data platform assembly daily record is stated, wherein, the option of operation that preserves includes the title of journal file, day
The quantity of the store path of will file, the size of journal file and journal file.
Alternatively, described device also includes：
Daily record warning module：For when log content in the standardization daily record be present and do not meet default audit regulation,
Then generate corresponding safe early warning information.
From above technical scheme, a kind of log audit method and device provided in an embodiment of the present invention, to from major
The original log that data platform component collects is parsed, field standardization maps and action type and the operation of daily record
Thin item division processing, it is possible to achieve the initial log of each component is standardized in the big data platform that source, form differ；So
Afterwards, according to the audit requirement of big data security management and control, using corresponding audit regulation and analysis strategy, to each in big data platform
Daily record after the standardization of component carries out automation audit analysis, to determine the management of big data platform and component and data access
Whether operation meets technical specification of security and management requires.Compared with manual audit's mode, daily record provided in an embodiment of the present invention
Auditing method, audited by the standardization to big data platform assembly daily record with concentrating, can be to big data platform assembly
Operation carry out comprehensively and timely audit, quickly find potential safety hazard, positioning security problem.
It should be appreciated that the general description and following detailed description of the above are only exemplary and explanatory, not
Can the limitation present invention.
Brief description of the drawings
In order to illustrate more clearly of technical scheme, letter will be made to the required accompanying drawing used in embodiment below
Singly introduce, it should be apparent that, for those of ordinary skills, without having to pay creative labor,
Other accompanying drawings can also be obtained according to these accompanying drawings.
Fig. 1 is that a kind of deployment framework of Log Audit System to big data platform assembly provided in an embodiment of the present invention shows
It is intended to；
Fig. 2 is a kind of schematic flow sheet of log audit method provided in an embodiment of the present invention；
Fig. 3 is that the scene provided in an embodiment of the present invention that daily record parsing is carried out to original log and standardizes field mapping is shown
It is intended to；
Fig. 4 is the schematic flow sheet of another log audit method provided in an embodiment of the present invention；
Fig. 5 is a kind of structural representation of log audit device provided in an embodiment of the present invention.
Embodiment
Here exemplary embodiment will be illustrated in detail, its example is illustrated in the accompanying drawings.Following description is related to
During accompanying drawing, unless otherwise indicated, the same numbers in different accompanying drawings represent same or analogous key element.Following exemplary embodiment
Described in embodiment do not represent and the consistent all embodiments of the present invention.On the contrary, they be only with it is such as appended
The example of the consistent apparatus and method of some aspects being described in detail in claims, of the invention.
The characteristics of scale due to being limited by big data platform is big, component and nodes are numerous, results in relying on manual type
The Operation Log of big data platform is audited, is unable to reach automation and the requirement of complete audit.In view of this, it is of the invention
A kind of log audit method and device for big data component is proposed, its basic realization principle is：First with regular expressions
Formula and field standard mapping ruler, are standardized to the original log gathered from big data component, then, with reference to big
The audit requirement and audit strategy of data safety management and control, the daily record after standardization to big data platform assembly carry out audit point
Analysis, audited so as to realize big data platform assembly Operation Log standardization and concentrate.
Fig. 1 is that a kind of deployment framework of Log Audit System to big data platform assembly provided in an embodiment of the present invention shows
It is intended to.As shown in figure 1, the present embodiment is by taking the big data platform based on Hadoop as an example, in Hadoop distributed file systems
In HDFS, a HDFS cluster includes a name node NameNode and multiple back end DataNode.
NameNode is master server, the access and offer of maintaining file system NameSpace, specification client for file
Operation for file directory.DataNode is responsible for the memory space on storage node and the read-write requests from client.
Therefore, the present embodiment design log acquisition server 10 (such as includes HDFS, HBase, Hive, Sqoop with big data platform assembly
Deng) communication connection, for gathering the journal file stored in big data platform assembly；In addition, design log analysis server 20
Communicated to connect with log acquisition server 10, the daily record for being gathered to log acquisition server 10 is parsed and audited.
, below will be to log audit method provided in an embodiment of the present invention based on above-mentioned basic realization principle and system architecture
Describe in detail.Fig. 2 is a kind of schematic flow sheet of log audit method provided in an embodiment of the present invention, applied to Fig. 1's
In log analysis server 10 and log acquisition server 20, it is necessary to explanation, the log analysis service in the present embodiment
Device 10 and log acquisition server 20 can also be integrated in same equipment body.As shown in Fig. 2 this method specifically include it is as follows
Step：
Step S110：The original log collected from big data platform assembly is parsed, obtains the original log
In effective log field and the effectively property value corresponding to log field.
The characteristics of form differs be present for the original log from each component of big data platform, the present embodiment is examined in daily record
Before meter, parsing is standardized to Operation Log, to improve the speed of follow-up audit and validity.Specifically, it can use
The mode of matching regular expressions, is parsed to the original log collected from big data platform assembly and attribute is drawn.Example
Such as, by a series of spcial characters of regular expression rule matching algorithm, the match pattern of daily record is built, then according to matching
Pattern matches to original log, the variate-value in regular expression of being extracted after the match is successful, and by attribute variable and its category
Property value is stored, and then obtains property value corresponding to effective log field in original log and effective log field.
Fig. 3 is that the scene provided in an embodiment of the present invention that daily record parsing is carried out to original log and standardizes field mapping is shown
It is intended to.As shown in figure 3, the daily record sample in figure can obtain the analysis result in following table one after matching regular expressions：
Table one：
Step S120：According to preset field mapping ruler, to category corresponding to effective log field in the original log
Property value carry out said standard field and sort out, obtain initial log.
Fig. 3 is that the scene provided in an embodiment of the present invention that daily record parsing is carried out to original log and standardizes field mapping is shown
It is intended to.As shown in figure 3, if property value corresponding to effective log field of extracting is create, then respective action.
Step S130：The keyword in effective log field in the initial log, carries out the initial log
Action type and operate the division of thin item, obtain standardizing daily record.
For example, according to operational order mkdir, can be using the thin item of division operation as directory creating, completion operation subclass is HDFS
Data access, action type are data access；Operational order is start namenode, then operates thin item and start for component, behaviour
It is assembly operating to make subclass, and action type is platform maintenance.
By the processing of above three step, and then realize the Operation Log standardization of big data platform assembly.Big data
The standardization result of component Operation Log is exemplified below shown in table two：
Table two：
Step S140：According to default audit regulation and analysis strategy, the standardization daily record to the big data platform assembly
Audited.
Based on the big data platform assembly Operation Log after standardization, carried out according to corresponding audit regulation and audit strategy
Audit, audit strategy include access time, access locations, important catalogue file, key operation order, the operation frequency, operation always
Dictionary or the baselines such as amount, while corresponding audit regulation is designed to each audit point in audit strategy, after to standardization
Operation Log based on it is above-mentioned strategy audit point audit, it can be found that non-working time, non-common site, non-security prefecture etc.
Data access, and access that the important catalogue file frequency is abnormal, accesses the unlawful practices such as significant data file superthreshold, so as to
It was found that the and safety problem and hidden danger of positioning big data platform management operation and data access operation.
Further, to realize the quick audit to the numerous Operation Logs of big data platform assembly, the present embodiment also provides
Another log audit method, it is specific as follows：
Step S141：Action type and the thin item of operation in the standardization daily record of the big data platform assembly, choosing
Fixed corresponding log content audit point and audit regulation.
Action type and the thin item of operation in standardization daily record, the daily record that the daily record is matched from default audit storehouse are examined
Enumeration and audit regulation, and then selectively the partial content in daily record can be audited, improve the speed of log audit
Degree.
Step S142：According to log content audit point and audit regulation, in the daily record of the standardization daily record
Appearance is audited.
It can be seen by above-described embodiment, the log audit method that this implementation provides, by from major data platform component
The original log collected is parsed, field standardization maps and the action type of daily record and the thin item division processing of operation,
It can realize that the initial log of each component in the big data platform to differ in source, form is standardized；Then, according to big data
The audit requirement of security management and control, using corresponding audit regulation and analysis strategy, the standardization to each component in big data platform
Daily record afterwards carries out automation audit analysis, to determine whether the management of big data platform and component and data access operation meet
Technical specification of security and management require.Audited by the standardization to big data platform assembly daily record with concentrating, can be right
The operation of big data platform assembly is carried out comprehensively and timely audited, and quickly finds potential safety hazard, positioning security problem.
To realize that collection comprehensively is realized in the daily record to big data platform assembly, audit coverage is improved, the present embodiment exists
Also the Operation Log of big data platform assembly is configured before log collection.Fig. 4 is another kind provided in an embodiment of the present invention
The schematic flow sheet of log audit method.As shown in figure 4, this method specifically comprises the following steps：
Step S210：Saving options for the big data platform assembly daily record is opened, and the big data platform is set
The preservation option of operation of component daily record, wherein, the storage for preserving option of operation and including the title, journal file of journal file
The quantity in path, the size of journal file and journal file.
Due to some daily records of big data platform assembly be give tacit consent to it is externally closing or without log recording, e.g., HDFS
Audit log records all HDFS requests, commonly writes in NameNode daily record, and acquiescence is to close.To realize to adopting comprehensively
Collection, the present embodiment also configure to big data component Operation Log, open the preservation choosing of the big data platform assembly daily record
, for follow-up acquisition server process log collection.
In addition, it can accurately and effectively collect the day that big data platform assembly preserved in order to facilitate acquisition server
Will,
The present embodiment also, sets the preservation option of operation of the big data platform assembly daily record, wherein, it is described to preserve operation
Option includes the title of journal file, the store path of journal file, the size of journal file and the quantity of journal file.
When preserving option of operation setting, the daily record of big data component writes journal file in a manner of additional, when one
Journal file reaches the file size of setting, generates next journal file, when the journal file number of generation reaches setting
During the latest document number of reservation, according to the generation time sequencing deleting history journal file of file.During in view of log collection
The memory cost and collecting efficiency during file are opened, the setting of individual log file is unsuitable excessive, such as suggests the big of single file
It is small to be configured to 256M, while disk expense is locally stored in view of daily record, journal file preserves number and should not configured excessively, such as
Proposed arrangement 20, such preservation option of operation are set, and have taken into account resource overhead and collecting efficiency.
Below with HDFS Operation Log examples of configurations：
What the service output that the HDFS daily records system such as including NameNode, DataNode, ResourceManage carries came
Daily record, acquiescence are stored under $ { HADOOP_HOME }/logs catalogues.HDFS audit logs record all HDFS requests, generally write
In the daily record for entering NameNode, acquiescence is to close.Pass through $ { HADOOP_HOME }/etc/hadoop/
The size for opening audit and configuration log file and the number of files of preservation etc., phase are set in log4j.properties property files
Config option is answered as shown in following table three
Table three：
And for example, it is without, it is necessary to manually add that MapReduce audits are default in log4j.properties configurations
Log4j.logger.org.apache.hadoop.yarn.server.resourcemanag er.RMAuditLogger=$
{ mapred.audit.logger }, so that MapReduce Operation Logs are opened.
Accordingly, daily record output is in the logs/mapred-audit.log files of resourcemanager main frames, day
Will form is as shown in following table four：
Table four：
Further, for the ease of log management and follow-up log collection, deposited in the present embodiment to journal file
When storing up the setting in path, node the depositing as journal file of the big data platform assembly interior joint minimum number is preferably chosen
Store up path.
It can also be stored in for example, HDFS audit logs can be stored on datanode on namenode, still
Datanode nodes are numerous, and daily record is distributed very scattered unsuitable collection, and namenode nodes are few, are more suitable for preserving hdfs audits
Daily record is in order to log management and collection.Therefore, when storing path is set, modification $ { HADOOP_HOME }/etc/hadoop/
Hdfs.audit.logger configures ExportHADOOP in HADOOP_NAMENODE_OPTS in hadoop-env.sh
NAMENODE OPTS=
...-D-Dhdfs.audit.logger=$ { HDFS_AUDIT_LOGGER:-INFO,RFAAUDIT}$HADOOP_
NAMENODE_OPTS, HDFS audit logs are made to be stored on namenode main frames.
Accordingly, daily record output namenode main frame logs/hdfs-audit.log, journal format is as shown in Table 5：
Table five：
Step S220：The original log collected from big data platform assembly is parsed, obtains the original log
In effective log field and the effectively property value corresponding to log field.
Wherein, when carrying out the log collection of big data platform assembly, daily record that big data platform assembly is stored can be with
Log collection probe is actively sent to by syslog protocol configurations, or, the big number of incremental crawler by the way of FTP/SFTP
Realized according to the operation log recording file of component.
On above two acquisition mode, the mode of syslog collections is ageing very high, and daily record passes through syslog once producing
Mode issue log acquisition server, may be considered substantially in real time, but syslog be based on udp agreements, it is restricted
Influenceed in network environment etc., might have the situation of daily record packet loss.By contrast, FTP/SFTP is Transmission Control Protocol, can be reliably high
Effect ground transmission data, even if exception occurs in network connection, can also re-establish connection, continue to gather at breakpoint.Two kinds are adopted
Another difference of mode set is exactly that syslog collections passively receive for acquisition component, and FTP is acquisition component
Actively initiate, more controllable and management, therefore the present embodiment incremental crawler daily record preferably by the way of FTP/SFTP.
Step S230：According to preset field mapping ruler, to category corresponding to effective log field in the original log
Property value carry out said standard field and sort out, obtain initial log.
Step S240：The keyword in effective log field in the initial log, carries out the initial log
Action type and operate the division of thin item, obtain standardizing daily record.
Step S250：According to default audit regulation and analysis strategy, the standardization daily record to the big data platform assembly
Audited.
Further, to realize the automation alarm to auditing result, the present embodiment is in step S260 to the big data
After the standardization daily record of platform assembly is audited, also comprise the following steps：
Step S260：When log content in the standardization daily record be present and do not meet default audit regulation, then phase is generated
The safe early warning information answered.
Specifically, can be according to the Non-Compliance in the daily record and the category of the big data platform assembly in the daily record institute source
Property information, generates corresponding safe early warning information.
Further, the safe early warning information of generation can also be sent to corresponding safe early warning platform, to pass through peace
The page of full early warning platform enters row information displaying.Or can be pressed in a manner of figure, form table etc. Asset Type, operation system,
Warning information is presented in the various dimensions such as assets director, and can combine the systems such as short message, mail, and the safe early warning information is sent
To default terminal, to notify the assets director of the compromised equipment, so that assets director can be informed in time, enter
And it can be threatened for accurate, quick exclusion and the best opportunity is provided.
From such scheme, the present embodiment is by the daily record of automatic concentrated collection big data platform assembly, using flexible
Standardized way carry out Operation Log standardization, and according to analysis rule and warning strategies realize big data platform assembly operate
Automation audit, from audit coverage, systematic function and ageing etc. General Promotion.
Based on above-mentioned log audit method, the embodiment of the present invention additionally provides a kind of log audit device.Fig. 5 is the present invention
A kind of structural representation for log audit device that embodiment provides.Include as shown in figure 5, the device has：
Original log parsing module 510：For being parsed to the original log collected from big data platform assembly, obtain
Effective log field and the property value effectively corresponding to log field into the original log；
Standardize field classifying module 520：For according to preset field mapping ruler, to effective in the original log
Property value corresponding to log field carries out said standard field and sorted out, and obtains initial log；
Action type division module 530：For the keyword in effective log field in the initial log, enter
The action type of the row initial log and the thin item division of operation, obtain standardizing daily record；
Standardize log audit module 540：For the default audit regulation of basis and analysis strategy, to the big data platform
The standardization daily record of component is audited.
To realize the quick audit to the numerous Operation Logs of big data platform assembly, above-mentioned standard log audit module
540 can include：
Audit strategy chooses submodule 541：For the operation in the standardization daily record according to the big data platform assembly
Type and the thin item of operation, select corresponding log content audit point and audit regulation；
Log content audit submodule 542：For according to the log content audit point and audit regulation, to the mark
The log content of standardization daily record is audited.
To realize that collection comprehensively is realized in the daily record to big data platform assembly, audit coverage is improved, the present embodiment carries
The log audit device of confession also includes：
Logging option setup module 550：For opening saving options for the big data platform assembly daily record, and set
The preservation option of operation of the big data platform assembly daily record, wherein, the title of the preservation option of operation including journal file,
The quantity of the store path of journal file, the size of journal file and journal file.
To realize the automation alarm to auditing result, the log audit device that the present embodiment provides also includes：
Daily record warning module 560：For log content when in the standardization daily record be present and do not meet default audit regulation
When, then generate corresponding safe early warning information.
The log audit device that the present embodiment provides, by the daily record of automatic concentrated collection big data platform assembly, use
Flexible standardized way carries out Operation Log standardization, and realizes big data platform assembly according to analysis rule and warning strategies
The automation audit of operation, from audit coverage, systematic function and ageing etc. General Promotion.
Each embodiment in this specification is described by the way of progressive, identical similar portion between each embodiment
Divide mutually referring to what each embodiment stressed is the difference with other embodiment.Especially for system or
For system embodiment, because it is substantially similar to embodiment of the method, so describing fairly simple, related part is referring to method
The part explanation of embodiment.System and system embodiment described above is only schematical, wherein as separation
The unit of part description can be or may not be it is physically separate, can be as the part that unit is shown or
It can not be physical location, you can with positioned at a place, or can also be distributed on multiple NEs.Can be according to reality
Border needs to select some or all of module therein to realize the purpose of this embodiment scheme.Those of ordinary skill in the art
Without creative efforts, you can to understand and implement.
It the above is only the embodiment of the present invention, it is noted that come for those skilled in the art
Say, under the premise without departing from the principles of the invention, some improvements and modifications can also be made, these improvements and modifications also should be regarded as
Protection scope of the present invention.
Claims (10)
- A kind of 1. log audit method, it is characterised in that methods described includes：The original log collected from big data platform assembly is parsed, obtains effective daily record word in the original log Section and the property value effectively corresponding to log field；According to preset field mapping ruler, affiliated mark is carried out to property value corresponding to effective log field in the original log Standardization field is sorted out, and obtains initial log；The keyword in effective log field in the initial log, action type and the behaviour for carrying out the initial log Make thin item division, obtain standardizing daily record；According to default audit regulation and analysis strategy, the standardization daily record to the big data platform assembly is audited.
- 2. according to the method for claim 1, it is characterised in that according to default audit regulation and analysis strategy, to described big The standardization daily record of data platform component is audited, including：Action type and the thin item of operation in the standardization daily record of the big data platform assembly, are selected in corresponding daily record Hold audit point and audit regulation；According to log content audit point and audit regulation, the log content of the standardization daily record is audited.
- 3. according to claim 1 method, it is characterised in that the original log that is collected from big data platform assembly is carried out Before parsing, methods described also includes：Saving options for the big data platform assembly daily record is opened, and the preservation of the big data platform assembly daily record is set Option of operation, wherein, the preservation option of operation includes the title of journal file, the store path of journal file, journal file Size and journal file quantity.
- 4. according to claim 3 method, it is characterised in that the choosing method of the store path of the journal file includes：Choose store path of the node of the big data platform assembly interior joint minimum number as journal file.
- 5. according to the method for claim 1, it is characterised in that the collection side of the original log of the big data platform assembly Formula includes：By syslog agreements, the original log that the big data platform assembly is sent is obtained；OrBy way of FTP/SFTP, the original log of its preservation is gathered from the big data platform assembly.
- 6. according to the method for claim 1, it is characterised in that the standardization daily record to the big data platform assembly is carried out After audit, methods described also includes：When log content in the standardization daily record be present and do not meet default audit regulation, then corresponding safe early warning letter is generated Breath.
- 7. a kind of log audit device, it is characterised in that described device includes：Original log parsing module：For being parsed to the original log collected from big data platform assembly, obtain described Effective log field and the property value effectively corresponding to log field in original log；Standardize field classifying module：For according to preset field mapping ruler, to effective daily record word in the original log Property value corresponding to section carries out said standard field and sorted out, and obtains initial log；Action type division module：For the keyword in effective log field in the initial log, described in progress The action type of initial log and the thin item division of operation, obtain standardizing daily record；Standardize log audit module：For the default audit regulation of basis and analysis strategy, to the big data platform assembly Standardization daily record is audited.
- 8. device according to claim 7, it is characterised in that the standardization log audit module includes：Audit strategy chooses submodule：For the action type in the standardization daily record according to the big data platform assembly and behaviour Make thin item, select corresponding log content audit point and audit regulation；Log content audit submodule：For according to the log content audit point and audit regulation, to the standardization day The log content of will is audited.
- 9. device according to claim 7, it is characterised in that described device also includes：Logging option setup module：For opening saving options for the big data platform assembly daily record, and set described big The preservation option of operation of data platform component daily record, wherein, the option of operation that preserves includes the title of journal file, daily record text The quantity of the store path of part, the size of journal file and journal file.
- 10. device according to claim 7, it is characterised in that described device also includes：Daily record warning module：For when log content in the standardization daily record be present and do not meet default audit regulation, then giving birth to Into corresponding safe early warning information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710994900.6A CN107818150B (en) | 2017-10-23 | 2017-10-23 | Log auditing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710994900.6A CN107818150B (en) | 2017-10-23 | 2017-10-23 | Log auditing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107818150A true CN107818150A (en) | 2018-03-20 |
| CN107818150B CN107818150B (en) | 2021-11-26 |
Family
ID=61607466
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710994900.6A Active CN107818150B (en) | 2017-10-23 | 2017-10-23 | Log auditing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107818150B (en) |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108768929A (en) * | 2018-04-09 | 2018-11-06 | 平安科技（深圳）有限公司 | The analytic method and storage medium of electronic device, reference feedback message |
| CN108959659A (en) * | 2018-08-14 | 2018-12-07 | 杭州安恒信息技术股份有限公司 | A kind of log access parsing method and system of big data platform |
| CN109040110A (en) * | 2018-08-31 | 2018-12-18 | 新华三信息安全技术有限公司 | A kind of outgoing behavioral value method and device |
| CN109325009A (en) * | 2018-09-19 | 2019-02-12 | 亚信科技（成都）有限公司 | The method and device of log parsing |
| CN109656894A (en) * | 2018-11-13 | 2019-04-19 | 平安科技（深圳）有限公司 | Log standardization storage method, device, equipment and readable storage medium storing program for executing |
| CN109885543A (en) * | 2018-12-24 | 2019-06-14 | 航天信息股份有限公司 | Log processing method and device based on big data cluster |
| CN110109809A (en) * | 2019-04-08 | 2019-08-09 | 武汉思普崚技术有限公司 | According to the method and apparatus of syslog test log audit function |
| CN110515792A (en) * | 2019-07-23 | 2019-11-29 | 平安科技（深圳）有限公司 | Monitoring method, device and computer equipment based on web edition task management platform |
| CN110764971A (en) * | 2019-10-30 | 2020-02-07 | 杭州安恒信息技术股份有限公司 | Auxiliary database operation and maintenance auditing method and device and electronic equipment |
| CN111339050A (en) * | 2018-12-03 | 2020-06-26 | 国网宁夏电力有限公司信息通信公司 | Centralized security audit method and system based on big data platform |
| CN112347165A (en) * | 2019-08-08 | 2021-02-09 | 腾讯科技（深圳）有限公司 | Log processing method and device, server and computer readable storage medium |
| CN112346938A (en) * | 2019-08-08 | 2021-02-09 | 腾讯科技（深圳）有限公司 | Operation auditing method and device, server and computer readable storage medium |
| CN112347066A (en) * | 2019-08-08 | 2021-02-09 | 腾讯科技（深圳）有限公司 | Log processing method and device, server and computer readable storage medium |
| CN112636957A (en) * | 2020-12-11 | 2021-04-09 | 微医云(杭州)控股有限公司 | Early warning method and device based on log, server and storage medium |
| CN113111037A (en) * | 2021-04-30 | 2021-07-13 | 杭州远石科技有限公司 | Log audit warning method, device and storage medium |
| CN113792076A (en) * | 2021-09-17 | 2021-12-14 | 甘肃同兴智能科技发展有限责任公司 | Data auditing system |
| CN114338352A (en) * | 2021-12-31 | 2022-04-12 | 南通机敏软件科技有限公司 | Audit log configuration and analysis method, storage medium and processor |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073579A (en) * | 2011-01-24 | 2011-05-25 | 复旦大学 | Method for merging and optimizing audit events of Linux file system |
| WO2012155455A1 (en) * | 2011-05-13 | 2012-11-22 | 中兴通讯股份有限公司 | Log analysis method and system based on web platform |
| CN104636494A (en) * | 2015-03-04 | 2015-05-20 | 浪潮电子信息产业股份有限公司 | Spark-based log auditing and reversed checking system for big data platforms |
| CN104717085A (en) * | 2013-12-16 | 2015-06-17 | 中国移动通信集团湖南有限公司 | Log parsing method and device |
| CN106815125A (en) * | 2015-12-02 | 2017-06-09 | 阿里巴巴集团控股有限公司 | A kind of log audit method and platform |
| CN107147639A (en) * | 2017-05-08 | 2017-09-08 | 国家电网公司 | A kind of actual time safety method for early warning based on Complex event processing |
| CN107273267A (en) * | 2017-06-09 | 2017-10-20 | 环球智达科技（北京）有限公司 | Log analysis method based on elastic components |
-
2017
- 2017-10-23 CN CN201710994900.6A patent/CN107818150B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073579A (en) * | 2011-01-24 | 2011-05-25 | 复旦大学 | Method for merging and optimizing audit events of Linux file system |
| WO2012155455A1 (en) * | 2011-05-13 | 2012-11-22 | 中兴通讯股份有限公司 | Log analysis method and system based on web platform |
| CN104717085A (en) * | 2013-12-16 | 2015-06-17 | 中国移动通信集团湖南有限公司 | Log parsing method and device |
| CN104636494A (en) * | 2015-03-04 | 2015-05-20 | 浪潮电子信息产业股份有限公司 | Spark-based log auditing and reversed checking system for big data platforms |
| CN106815125A (en) * | 2015-12-02 | 2017-06-09 | 阿里巴巴集团控股有限公司 | A kind of log audit method and platform |
| CN107147639A (en) * | 2017-05-08 | 2017-09-08 | 国家电网公司 | A kind of actual time safety method for early warning based on Complex event processing |
| CN107273267A (en) * | 2017-06-09 | 2017-10-20 | 环球智达科技（北京）有限公司 | Log analysis method based on elastic components |
Cited By (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108768929B (en) * | 2018-04-09 | 2021-04-13 | 平安科技（深圳）有限公司 | Electronic device, credit investigation feedback message analysis method and storage medium |
| CN108768929A (en) * | 2018-04-09 | 2018-11-06 | 平安科技（深圳）有限公司 | The analytic method and storage medium of electronic device, reference feedback message |
| CN108959659A (en) * | 2018-08-14 | 2018-12-07 | 杭州安恒信息技术股份有限公司 | A kind of log access parsing method and system of big data platform |
| CN108959659B (en) * | 2018-08-14 | 2021-09-07 | 杭州安恒信息技术股份有限公司 | Log access analysis method and system for big data platform |
| CN109040110A (en) * | 2018-08-31 | 2018-12-18 | 新华三信息安全技术有限公司 | A kind of outgoing behavioral value method and device |
| CN109325009A (en) * | 2018-09-19 | 2019-02-12 | 亚信科技（成都）有限公司 | The method and device of log parsing |
| CN109325009B (en) * | 2018-09-19 | 2021-11-30 | 亚信科技（成都）有限公司 | Log analysis method and device |
| CN109656894A (en) * | 2018-11-13 | 2019-04-19 | 平安科技（深圳）有限公司 | Log standardization storage method, device, equipment and readable storage medium storing program for executing |
| CN111339050A (en) * | 2018-12-03 | 2020-06-26 | 国网宁夏电力有限公司信息通信公司 | Centralized security audit method and system based on big data platform |
| CN111339050B (en) * | 2018-12-03 | 2023-07-18 | 国网宁夏电力有限公司信息通信公司 | Centralized security audit method and system based on big data platform |
| CN109885543A (en) * | 2018-12-24 | 2019-06-14 | 航天信息股份有限公司 | Log processing method and device based on big data cluster |
| CN110109809A (en) * | 2019-04-08 | 2019-08-09 | 武汉思普崚技术有限公司 | According to the method and apparatus of syslog test log audit function |
| CN110515792B (en) * | 2019-07-23 | 2022-11-25 | 平安科技（深圳）有限公司 | Monitoring method and device based on web version task management platform and computer equipment |
| CN110515792A (en) * | 2019-07-23 | 2019-11-29 | 平安科技（深圳）有限公司 | Monitoring method, device and computer equipment based on web edition task management platform |
| CN112347066A (en) * | 2019-08-08 | 2021-02-09 | 腾讯科技（深圳）有限公司 | Log processing method and device, server and computer readable storage medium |
| CN112346938A (en) * | 2019-08-08 | 2021-02-09 | 腾讯科技（深圳）有限公司 | Operation auditing method and device, server and computer readable storage medium |
| CN112347165A (en) * | 2019-08-08 | 2021-02-09 | 腾讯科技（深圳）有限公司 | Log processing method and device, server and computer readable storage medium |
| CN112347066B (en) * | 2019-08-08 | 2023-10-13 | 腾讯科技（深圳）有限公司 | Log processing method and device, server and computer readable storage medium |
| CN112347165B (en) * | 2019-08-08 | 2023-11-03 | 腾讯科技（深圳）有限公司 | Log processing method and device, server and computer readable storage medium |
| CN110764971A (en) * | 2019-10-30 | 2020-02-07 | 杭州安恒信息技术股份有限公司 | Auxiliary database operation and maintenance auditing method and device and electronic equipment |
| CN112636957A (en) * | 2020-12-11 | 2021-04-09 | 微医云(杭州)控股有限公司 | Early warning method and device based on log, server and storage medium |
| CN112636957B (en) * | 2020-12-11 | 2023-02-21 | 微医云(杭州)控股有限公司 | Early warning method and device based on log, server and storage medium |
| CN113111037A (en) * | 2021-04-30 | 2021-07-13 | 杭州远石科技有限公司 | Log audit warning method, device and storage medium |
| CN113792076A (en) * | 2021-09-17 | 2021-12-14 | 甘肃同兴智能科技发展有限责任公司 | Data auditing system |
| CN114338352A (en) * | 2021-12-31 | 2022-04-12 | 南通机敏软件科技有限公司 | Audit log configuration and analysis method, storage medium and processor |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107818150B (en) | 2021-11-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107818150A (en) | A kind of log audit method and device | |
| US10917319B2 (en) | MDL-based clustering for dependency mapping | |
| Fire et al. | Organization mining using online social networks | |
| Forester et al. | Modeling human factors that affect the loss of biodiversity | |
| US10140453B1 (en) | Vulnerability management using taxonomy-based normalization | |
| EP2088711B1 (en) | A log analyzing method and system based on distributed compute network | |
| CN107943668A (en) | Computer server cluster daily record monitoring method and monitor supervision platform | |
| US20170109657A1 (en) | Machine Learning-Based Model for Identifying Executions of a Business Process | |
| US9600503B2 (en) | Systems and methods for pruning data by sampling | |
| CN107577805A (en) | A kind of business service system towards the analysis of daily record big data | |
| CN107409126A (en) | System and method for protecting enterprise computing environment safety | |
| CN106815125A (en) | A kind of log audit method and platform | |
| US20150032725A1 (en) | Systems and methods for efficient data ingestion and query processing | |
| US20170109636A1 (en) | Crowd-Based Model for Identifying Executions of a Business Process | |
| CN107733902A (en) | A kind of monitoring method and device of target data diffusion process | |
| CN109213919A (en) | A kind of information technology consultative service system Internet-based | |
| CN107169143A (en) | A kind of efficient magnanimity public sentiment data message trunking matching process | |
| CN113626447B (en) | Civil aviation data management platform and method | |
| CN103067355A (en) | Network guard method and system based on domain name | |
| Miranda et al. | Measuring and modeling software vulnerability security advisory platforms | |
| CN106326280A (en) | Data processing method, apparatus and system | |
| US11463483B2 (en) | Systems and methods for determining effectiveness of network segmentation policies | |
| JP2024507797A (en) | Standardization in the context of data integration | |
| Sahin et al. | Streaming event detection in microblogs: Balancing accuracy and performance | |
| CN112346938B (en) | Operation auditing method and device, server and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |