CN113420032A - Classification storage method and device for logs - Google Patents
Classification storage method and device for logs Download PDFInfo
- Publication number
- CN113420032A CN113420032A CN202110820989.0A CN202110820989A CN113420032A CN 113420032 A CN113420032 A CN 113420032A CN 202110820989 A CN202110820989 A CN 202110820989A CN 113420032 A CN113420032 A CN 113420032A
- Authority
- CN
- China
- Prior art keywords
- log
- logs
- storing
- alarm
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 67
- 238000012545 processing Methods 0.000 claims abstract description 78
- 238000005516 engineering process Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 238000012216 screening Methods 0.000 description 22
- 238000004458 analytical method Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 7
- 238000002955 isolation Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 230000006399 behavior Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 230000002159 abnormal effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 230000003139 buffering effect Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000003780 insertion Methods 0.000 description 3
- 230000037431 insertion Effects 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 241000700605 Viruses Species 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application provides a classified storage method and a classified storage device for logs, wherein the method comprises the following steps: classifying the collected logs according to key fields in the logs; and respectively sending the classified different types of logs to corresponding log processing programs for parallel processing, and storing the logs into a database. According to the technical scheme, different types of logs are respectively sent to corresponding log processing programs for parallel processing, so that the efficiency of storing the logs into the database can be improved; in addition, different types of logs are respectively put in storage, and the readability and the operability of the whole log system can be improved.
Description
Technical Field
The present application relates to the field of information technologies, and in particular, to a method and an apparatus for storing logs in a classified manner, a computer device, and a computer-readable storage medium.
Background
At present, the service of data synchronization through the security isolation gatekeeper is in diversified form development, and the data exchange requirements are increasingly raised, such as video data exchange, WEB service data exchange and the like, and the security isolation gatekeeper has the typical characteristics of large data exchange amount and timely and efficient requirements. How to check the integrity of data transmitted through a network gate becomes important for constructing a complete log auditing system, so that the log information related to the service is ensured to be kept in the system, and the requirements of efficient and real-time recording are met.
However, the service throughput of the gatekeeper is increased day by day, the log quantity of various services and systems is also multiplied, and the time consumed for storing and warehousing a large amount of logs is increased more and more.
Therefore, in order to meet the auditing requirement of the current real-time service, it is necessary to provide a scheme for efficiently and quickly storing the logs into the database, so that the method can be applied to a network gate and other scenes with a large amount of log storage requirements.
Disclosure of Invention
The application aims to provide a log classified storage method and device, computer equipment and a computer readable storage medium, which are used for improving the efficiency of storing logs into a database, so that the log classified storage method and device are suitable for occasions with a large number of log storage requirements, such as a gatekeeper.
One aspect of the embodiments of the present application provides a method for storing logs in a classified manner, including:
classifying the collected logs according to key fields in the logs;
and respectively sending the classified different types of logs to corresponding log processing programs for parallel processing, and storing the logs into a database.
Optionally, the different types of logs specifically include: at least one of a log of a service type, a log of an alarm type, and a log of a kernel type; and
the database includes at least one of the following data tables: the system comprises a service log data table for storing service type logs, an alarm log data table for storing alarm type logs and a kernel log data table for storing kernel type logs.
Optionally, when the log is a service type log, sending the log to a service log processing program for processing and then storing the processed log into the service log data table; the service type logs are divided into a plurality of subcategories, and the service log data tables are multiple and respectively correspond to each subcategory of the service type logs;
the method for processing the log by the service log processing program comprises the following steps: after the logs with correct formats are screened out by utilizing a regular matching technology, inserting the screened logs into a pre-constructed annular buffer queue;
reading logs one by one from the circular buffer queue;
aiming at each currently read log, identifying the sub-category of the log, and caching the log in a log storage queue corresponding to the sub-category according to the identified sub-category;
and aiming at each sub-category, when the number of the logs in the log storage queue corresponding to the sub-category reaches a set number index, inserting the logs in the log storage queue into a service log data table corresponding to the sub-category in the database in batches, and emptying the log storage queue.
Optionally, after the regular matching technology is used to screen out the logs with the correct format, the screened logs are inserted into a pre-constructed circular buffer queue, which specifically includes:
acquiring a log data block with a set size through a block analyzer, performing message format check on the acquired log data block, discarding unconventional log information in the log data block, and splitting the log data block into log message information with minimum information length;
distributing the split log message information of each row to a plurality of parallel running row resolvers through the block resolvers, and carrying out further rule check by the row resolvers:
and checking whether the information of a plurality of necessary fields in the input log message information of each line is in compliance or not through the line analyzer, further discarding the log message information which is not in compliance, and inserting the log message information which is in compliance into the ring buffer queue.
Optionally, the checking, by the line parser, whether information of a plurality of necessary fields in the input log message information of each line is compliant specifically includes:
the method comprises the steps that sub-categories of log message information input within a period of time are divided through a line analyzer; and aiming at the divided log message information of each sub-category, carrying out batch rule check on the log message information of the sub-category by using an SQL statement block corresponding to the sub-category.
Optionally, when the log is an alarm type log, sending the alarm type log to an alarm log processing program for processing and then storing the processed alarm type log into the alarm log data table; the method for processing the log by the alarm log processing program comprises the following steps:
the alarm log processing program compares the information abstract of the currently input log with the information abstract of each log stored in the first hash table; if the comparison result is inconsistent, storing the currently input log into a first hash table and a first cache queue; otherwise:
further comparing the status identification bits of the two logs with the same information abstract; if the status identification bits of the log and the hash table are different, storing the currently input log into a first cache queue, and updating the status identification bit of the corresponding log in a first hash table according to the status identification bit of the currently input log;
if the status identification bits of the two logs are the same, the timestamps of the two logs are further compared; if the difference between the timestamps of the log and the hash table is larger than a set value, storing the log into a first cache queue, and updating the timestamp of the corresponding log in a first hash table according to the timestamp of the currently input log;
and storing the log in the first cache queue into an alarm log data table in the database.
Optionally, when the log is a kernel-type log, sending the log to a kernel log processing program for processing and then storing the processed log into the kernel log data table; the method for processing the logs by the kernel log processing program comprises the following steps:
comparing the protocol information of the currently input log with the protocol information of each log stored in the second hash table; if the comparison result is inconsistent, storing the log into a second hash table and a second cache queue; otherwise:
further comparing the status identification bits of two logs with the same protocol information; if the status identification bits of the log and the hash table are different, storing the log into a second cache queue, and updating the status identification bit of the corresponding log in a second hash table according to the currently input status identification bit of the log;
if the status identification bits of the two logs are the same, the timestamps of the two logs are further compared; if the difference between the timestamps of the log and the hash table is larger than a set value, storing the log into a second cache queue, and updating the timestamp of the corresponding log in a second hash table according to the timestamp of the currently input log;
and storing the logs in the second cache queue to a kernel log data table in the database.
An aspect of an embodiment of the present application further provides a log classification storage apparatus, including:
the log collection module is used for collecting logs and classifying the collected logs according to key fields in the logs;
and the log processing modules correspond to the classified logs of different types respectively and are used for processing the logs of the corresponding types and storing the processed logs into the database.
An aspect of the embodiments of the present application further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the log classification storage method.
An aspect of the embodiments of the present application further provides a computer-readable storage medium, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the log classification storage method.
According to the classified storage method, the computer equipment and the computer readable storage medium of the log, the collected log is classified according to key fields in the log; and respectively sending the classified different types of logs to corresponding log processing programs for parallel processing, and storing the logs into a database. Therefore, different types of logs are respectively sent to corresponding log processing programs for parallel processing, the efficiency of storing the logs into a database can be improved, and the method and the device are suitable for scenes with a large number of log storage requirements, such as a gatekeeper. In addition, different types of logs are respectively put in storage, and the readability and the operability of the whole log system can be improved.
Preferably, when the logs with correct formats are screened, the block analyzer and the line analyzer are adopted for screening the logs, the block analysis processing speed is high, the information source can be quickly obtained for preliminary screening, the line analysis granularity is accurate, the information source can be bound to multiple cores for calculation processing, the log screening efficiency is integrally improved, and the log warehousing efficiency can be further improved.
Preferably, the chunk parser is a production thread and the line parser is a plurality of consumption threads. The design mode utilizes the characteristics of a multi-core CPU system and shares the analysis calculation to each core, thereby improving the analysis efficiency, and also improving the efficiency of log screening and the efficiency of log storage.
Preferably, the line parser can perform centralized parsing, checking and buffering on the same sub-categories accumulated within a period of time, and avoid switching SQL sentences back and forth, so that the efficiency of screening logs by the line parser is greatly improved, and the efficiency of warehousing logs can also be improved.
In addition, the logs of different sub-categories are stored in the database in a table mode, and the logs of the same sub-category are stored in the same data table, so that the readability and the operability of the whole log system are improved.
Preferably, when the alarm/kernel logs are processed, the similar alarm/kernel logs can be merged according to a state edge triggering mode due to the comparison of information summary/protocol information and state identification bits; through the comparison of the time stamps, the redundant logs reported repeatedly within a period of time can be merged. After the redundant logs are merged, the number of the logs which are executed to be put in storage can be greatly reduced, and therefore the storage efficiency of the logs is improved.
Drawings
FIG. 1 is a schematic diagram illustrating a classified storage device of logs applied to a security isolation gatekeeper according to an embodiment of the present invention;
FIG. 2 is a block diagram schematically illustrating the internal structure of a sorting storage apparatus for implementing a log according to the present invention;
FIG. 3 is a flow chart of a method for processing a log by a service log processing program according to a first embodiment of the invention;
fig. 4 is a schematic diagram illustrating an internal structure of a service log processing module according to a first embodiment of the present invention;
FIG. 5 is a flow chart of a method for processing a log by an alarm log handler according to a second embodiment of the invention;
FIG. 6 is a flow chart of a method for processing logs by a kernel log handler according to a third embodiment of the present invention;
fig. 7 is a schematic diagram of a hardware architecture of a computer device suitable for implementing the classified storage method of the log according to a fourth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the descriptions relating to "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present application.
In the description of the present application, it should be understood that the numerical references before the steps do not identify the order of performing the steps, but merely serve to facilitate the description of the present application and to distinguish each step, and therefore should not be construed as limiting the present application.
The inventor of the application finds that different types of logs generally have different warehousing processing modes; logs of the same type often have approximately the same warehousing operation mode; therefore, in the technical scheme of the application, the collected logs are classified according to the key fields in the logs; and respectively sending the classified different types of logs to corresponding log processing programs for parallel processing, and storing the logs into a database. Therefore, different types of logs are respectively sent to corresponding log processing programs for parallel processing, the efficiency of storing the logs into a database can be improved, and the method and the device are suitable for scenes with a large number of log storage requirements, such as a gatekeeper. In addition, different types of logs are respectively put in storage, and the readability and the operability of the whole log system can be improved.
Fig. 1 shows a schematic diagram of a log classification storage device installed in a security isolation gatekeeper.
In the security isolation gatekeeper, file traffic, database traffic, and proxy traffic generate logs. The logs are sent to a log collector syslog-ng in a classification storage device of the logs by means of a pipeline file, a local network and the like.
The security isolation gatekeeper usually includes three types of logs, which are respectively a service type log (referred to as a service log for short), an alarm type log (referred to as an alarm log for short) and a kernel type log (referred to as a kernel log for short). The service log and the alarm log are generated by each large service module of the security isolation gatekeeper independently, and the format of the service log and the alarm log is optimized by dividing the data into fields on the basis of the syslog format which accords with the standard. That is, the data source of the alarm log is the same as that of the service log, and the data source is the log generated by the file service, the database service and the proxy service.
The kernel log is a log generated by the kernel system. The kernel log is generally the contents of the system itself such as start, stop, process abnormal information, etc., and the corresponding log will be generated when the gatekeeper is attacked by the network. The log processing is uniformly delivered to a log collector syslog-ng in the classification storage device for log collection.
In the log classifying and storing device, the collected logs are classified accurately according to the fields, then distributed to different log processing programs, and simultaneously stored in different data tables in the database, so that high concurrent processing logic of the logs is realized.
The log processing programs corresponding to the service type log (service log for short), the alarm type log (alarm log for short) and the kernel type log (kernel log for short) are a service log processing program, an alarm log processing program and a kernel log processing program, respectively.
Correspondingly, the database may also include at least one of the following data tables: the system comprises a service log data table for storing service type logs, an alarm log data table for storing alarm type logs and a kernel log data table for storing kernel type logs.
The classification storage device for the log provided by the application can comprise: the system comprises a log collection module and a plurality of log processing modules;
the log collection module may include the log collector syslog-ng, which is configured to collect logs and classify the collected logs according to key fields in the logs. Wherein the key field may be a rank field or a module field; for example, the log collection module may identify a level field in the log, and if the level field of the log is a normal level, the log is classified as a service type log (service log for short); if the level field of the log is specifically the alarm level, the log is classified as an alarm type log (alarm log for short); if the level field of the log is the kernel level, the log is classified as a kernel-type log (referred to as a kernel log for short).
And the log processing modules correspond to the classified logs of different types respectively and are used for processing the logs of the corresponding types and storing the processed logs into the database.
The internal structure of the classification storage device for logs provided by the present application can be as shown in fig. 2, and includes the following modules: a log collection module 201, a service log processing module 202, an alarm log processing module 203 and a kernel log processing module 204.
The service log processing module 202 corresponds to a service type log (referred to as a service log for short) and is configured to process the service log;
the alarm log processing module 203 is corresponding to a log of alarm types (called alarm log for short) and is used for processing the alarm log;
the kernel log processing module 204 corresponds to a kernel-type log (referred to as a kernel log for short) and is configured to process the kernel log.
Various embodiments are provided below, and various embodiments provided below can be used to implement the above-described scheme for sorted storage of logs.
Example one
Fig. 3 schematically shows a flowchart of a method for processing a log by a service log processing program according to a first embodiment of the present application.
As shown in fig. 3, a method for processing a service log by a service log processing program according to a first embodiment of the present application may include the following steps:
step S301: and for the service type logs, screening out logs with correct formats by utilizing a regular matching technology, and then inserting the screened logs into an annular buffer queue.
Specifically, the service log processing program mainly processes a system log, a management log, a service log and a tracking log; in this step, the service log processing process may first screen out dirty logs, such as logs with empty key field content and incorrect field format, by using a regular matching technique, and truncate the ultralong log content, so as to facilitate warehousing. The logs with correct format are inserted into a ring buffer queue which is constructed in advance; when the ring buffer queue is full, subsequent logs will be discarded.
Preferably, in this step, a log data block with a set size, for example, a log data block with a size of 512KB, 1MB or 10MB, may be obtained through the block parser, and a syslog message format check is performed on the obtained log data block, and after non-compliant log information in the log data block is discarded, the log data block is split into log message information with a minimum information length in a row, that is, log message information in a row;
the block parser distributes the split log message information of each row to a plurality of parallel running row parsers, and the row parsers perform further rule check: the line analyzer checks whether the information of a plurality of necessary fields in the line of log message information is in compliance or not aiming at the input log message information of each line, discards the log message information which is not in compliance and inserts the log message information which is in compliance into the annular buffer queue;
the block parser and the line parser are implemented by using a syntax interpreter respectively, and functionally are the difference between coarse-grained detection and fine-grained detection. The advantage of adopting block analyzer and line analyzer lies in, the fast information source that can acquire fast of block analysis processing rate carries out preliminary screening, and line analysis granularity is accurate can bind to and carry out the calculation processing on the multicore to improve the efficiency of log screening on the whole, just also can improve the efficiency that the log was put in storage.
A memory queue is arranged between the block parser and the row parser, the memory queue is mainly used for information transmission among threads, the block parser is a production thread, and the row parser is a plurality of consumption threads. The design mode utilizes the characteristics of a multi-core CPU system and shares the analysis calculation to each core, thereby improving the analysis efficiency, and also improving the efficiency of log screening and the efficiency of log storage. The memory queue supports a single-production-multi-consumption multithread processing mode, and a plurality of consumption threads can equally acquire messages output by the generation threads. With the concept of thread pool, the number of threads of the line parser can be configured in the initial loading stage of the program.
In general, the logs of traffic types can be divided into various sub-categories, for example, distinguished by the sub-categories of the traffic logs: web module logs, flow tracking logs, file tracking logs, database tracking logs and general service logs;
as a more preferable embodiment, the line parser classifies the message information of the log input in a period of time into sub-categories; the row parser maintains an SQL (Structured Query Language) statement block corresponding to each sub-type log for parsing and checking the sub-type log; and the row parser performs batch rule check on the log message information of each sub-category by using the SQL statement block corresponding to the sub-category aiming at the batch log message information of each sub-category. The line analyzer can perform centralized analysis, inspection and buffering on the same sub-categories accumulated in a period of time, and avoids switching SQL sentences back and forth, so that the efficiency of screening logs by the line analyzer is greatly improved, and the efficiency of warehousing the logs can also be improved.
Step S302: and reading logs from the circular buffer queue one by one.
The service log processing program can also read logs from the ring buffer queue one by one and use the read logs for warehousing.
The ring buffer queue is used for communication among multiple processes, and a multi-producer-multi-reader model can be realized by a memory sharing method; for example, in addition to reading the logs from the ring buffer queue during warehousing, the logs can also be read from the ring buffer queue for log snapshot, that is, real-time log information in the ring buffer queue is synchronized to each log snapshot. The log snapshot has the function of displaying the latest log information in real time, and has the advantages that the log snapshot is in a read-only mode, and the operation of acquiring the log by a single log snapshot does not block the log to be put in storage. The log snapshot can be used for the CLI viewing of a background manager and also can be used for the WebUI log audit viewing.
Step S303: and aiming at each currently read log, identifying the sub-category of the log, and caching the log in a log storage queue corresponding to the sub-category according to the identified sub-category.
Step S304: and aiming at each sub-category, when the number of the logs in the log storage queue corresponding to the sub-category reaches a set number index, inserting the logs in the log storage queue into a service log data table corresponding to the sub-category in the database in batches, and emptying the log storage queue.
Specifically, a plurality of service log data tables in the database respectively correspond to each subcategory of the service log; in this step, for each sub-category, when it is determined that the number of logs in the log storage queue corresponding to the sub-category reaches the set number index, the service log processing program inserts the logs in the log storage queue into the service log data table corresponding to the sub-category in the database in batches, and clears the log storage queue.
The logs of different sub-categories are stored in the database in a table mode, and the logs of the same sub-category are stored in the same data table, so that the readability and the operability of the whole log system are improved.
Corresponding to the method for processing the service log by the service log processing program, a service log processing module 202 provided in the first embodiment of the present application is specifically configured to, for a service-type log, filter out a log with a correct format by using a regular matching technique, and then insert the filtered log into a pre-constructed circular buffer queue; reading logs one by one from the circular buffer queue; aiming at each currently read log, identifying the sub-category of the log, and caching the log in a log storage queue corresponding to the sub-category according to the identified sub-category; and aiming at each sub-category, when the number of the logs in the log storage queue corresponding to the sub-category reaches a set number index, inserting the logs in the log storage queue into a service log data table corresponding to the sub-category in the database in batches, and emptying the log storage queue.
An internal structure of a service log processing module 202 provided in an embodiment of the present application is shown in fig. 4, and may include the following units: a log screening unit 401 and a log storage unit 402;
the log screening unit 401 is configured to, for logs of a service type, screen out logs with a correct format by using a regular matching technology, and then insert the screened logs into a pre-constructed circular buffer queue; preferably, the log filtering unit 401 may include the above block parser and line parser.
The log warehousing unit 402 is configured to read logs from the circular buffer queue one by one; aiming at each currently read log, identifying the sub-category of the log, and caching the log in a log storage queue corresponding to the sub-category according to the identified sub-category; and for each sub-category, when the number of the logs in the log storage queue corresponding to the sub-category reaches a set number index, inserting the logs in the log storage queue into a service log data table corresponding to the sub-category in the database in batches, and emptying the log storage queue.
According to the technical scheme, the block analyzer and the line analyzer are adopted for screening the logs, the block analysis processing speed is high, the information source can be rapidly acquired for preliminary screening, the line analysis granularity can be accurately bound to multiple cores for calculation processing, and therefore the efficiency of log screening is integrally improved, and the efficiency of log storage can be further improved.
Preferably, the chunk parser is a production thread and the line parser is a plurality of consumption threads. The design mode utilizes the characteristics of a multi-core CPU system and shares the analysis calculation to each core, thereby improving the analysis efficiency, and also improving the efficiency of log screening and the efficiency of log storage.
Preferably, the line parser can perform centralized parsing, checking and buffering on the same sub-categories accumulated within a period of time, and avoid switching SQL sentences back and forth, so that the efficiency of screening logs by the line parser is greatly improved, and the efficiency of warehousing logs can also be improved.
In addition, the logs of different sub-categories are stored in the database in a table mode, and the logs of the same sub-category are stored in the same data table, so that the readability and the operability of the whole log system are improved.
Example two
The second embodiment of the present application describes a scheme for processing an alarm log by an alarm log processing program.
Alarm logs are typically divided into eight subtypes: virus alarms, attack alarms, hardware exceptions, system exceptions, resource exceptions, configuration changes, log alarms, and policy alarms. Each seed type alarm log is individually identified by a type identification bit, and in addition, the status identification bit identifies the change of the state of the alarm log, such as a resource abnormal alarm log, taking a cpu alarm log as an example, the state identification bit is high (high)/medium (mid)/low (low); for example, the system abnormal alarm log takes a network card alarm log as an example, and the status flag bit is fault (fault)/recovery (recovery).
Fig. 5 is a flowchart schematically illustrating a method for processing a log by an alarm log processing program according to a second embodiment of the present application.
As shown in fig. 5, the method for processing an alarm log by an alarm log processing program according to the second embodiment of the present application may include the following steps:
and S501, screening the logs with the correct format by utilizing a regular matching technology for the logs with the alarm types.
In this step, the alarm log processing program screens out the log with the correct format by using the regular matching technology for the log of the alarm type, and the method for screening out the dirty log may be the same as the method for screening out the log in step S301 in fig. 3, and is not described here again.
Because the alarm log can be matched with an outgoing program to send alarm contents to users in time, such as short messages, mailboxes and the like, a plurality of alarm logs with the same contents can be sent within a period of time. Therefore, in order to improve the warehousing efficiency of the logs, the following steps are adopted to merge the screened logs with correct formats so as to greatly reduce the number of logs needing to be warehoused.
Step S502: the alarm log processing program compares the information abstract of the currently input log with the information abstract of each log stored in the first hash table; if the comparison result is inconsistent, executing the following step S503 to store the currently input log into the first hash table and the first cache queue; otherwise, the following step S504 is executed;
specifically, the msg (information) field of the currently input alarm log with the correct format may be hashed to extract a fixed-length information digest, where the information digest is a unique field of the alarm log. The digest has a function of being index information and compression information.
Comparing the information abstract of the alarm log with the correct format with the information abstract of each log stored in the hash table; if there is no log with the same information digest as the currently input alarm log in the hash table, executing the following step S503 to store the currently input log in the first hash table and the first cache queue; otherwise, the following step S504 is performed.
Step S503: and storing the currently input log into a first hash table and a first cache queue.
In this step, the currently input log is stored in the first hash table, and the currently input log is also stored in the first buffer queue to be put in storage.
Step S504: further comparing the status identification bits of the two logs with the same information abstract; if the status flag bits of the first hash table and the second hash table are different, executing step S505 to store the currently input log into the first cache queue, and updating the status flag bit of the corresponding log in the first hash table according to the status flag bit of the currently input log; if the status flag bits are the same, step S506 is executed.
Specifically, if there is a log with the same information digest as the currently input alarm log in the hash table, the status flag bits of the two logs with the same information digest are continuously compared in this step: extracting a state identification bit from an msg field of a currently input alarm log, comparing the state identification bit with the state identification bit of the log with the same information abstract, and comparing whether the state identification bit is changed and is changed from low to high or from fault to receiver; if the change occurs, the following step S505 is executed to store the currently input log into the first cache queue, and update the status flag of the log with the same information digest in the first hash table according to the status flag of the currently input log.
Step S505: storing the currently input log into a first cache queue, and updating a state identification bit of the corresponding log in a first hash table according to the state identification bit of the currently input log;
in this step, the currently input log is stored in a first cache queue to be put in storage, and the state identification bit of the log with the information abstract same as that of the currently input log in the first hash table is updated according to the state identification bit of the currently input log.
Step S506: further comparing the timestamps of the two logs; if the difference between the two timestamps is greater than the set value, executing step S507 to store the log into the first cache queue, and updating the timestamp of the corresponding log in the first hash table according to the timestamp of the currently input log; otherwise, go to step S508;
specifically, if a log with the same information abstract and unchanged identification bit as the information abstract of the currently input alarm log exists in the hash table, comparing the time stamp of the log with the time stamp of the currently input alarm log; if the difference between the two timestamps is greater than the set value, that is, the insertion time of the two timestamps exceeds the time length of the timer, step S507 is executed to store the log into the first cache queue, and the timestamp of the corresponding log in the first hash table is updated according to the timestamp of the currently input log.
Step S507: storing the currently input log into a first cache queue, and updating a timestamp of the corresponding log in a first hash table according to the timestamp of the currently input log;
in the step, the currently input log is stored in a first cache queue to prepare for storage, and the timestamp of the log which is the same as the summary of the log information and has unchanged state identification bits in a first hash table is updated according to the timestamp of the currently input log.
Step S508: the currently entered log is discarded.
In the step, merging processing of repeated logs is carried out; through the multiple judgments in the above steps, in this step, the logs which have the same information summary and the same state identification bit as the currently input logs and the time stamp difference within the set value range are determined to be stored in the first hash table, and then the currently input logs are discarded as the repeated logs, so that the repeated warehousing operation of multiple logs with the same content is avoided.
Step S509: and storing the log in the first cache queue into an alarm log data table in the database.
The log of the first buffer queue can be put into the above ring buffer queue as a snapshot while the database insertion operation is executed, and the principle is the same as that of the service log.
Therefore, by comparing the information abstract and the state identification bits, the similar alarm logs can be merged according to a state edge triggering mode; through the comparison of the time stamps, the redundant logs reported repeatedly within a period of time can be merged. After the redundant logs are merged, the number of the logs which are executed to be put in storage can be greatly reduced, and therefore the storage efficiency of the logs is improved.
Corresponding to the method for processing the alarm log by the alarm log processing program, the alarm log processing module 203 provided in the second embodiment of the present application is specifically configured to compare the information digest of the currently input log with the information digests of the logs stored in the first hash table; if the comparison result is inconsistent, storing the log into a first hash table and a first cache queue; otherwise: further comparing the status identification bits of the two logs with the same information abstract; if the status identification bits of the log and the hash table are different, storing the log into a first cache queue, and updating the status identification bit of the corresponding log in a first hash table according to the currently input status identification bit of the log; if the status identification bits of the two logs are the same, the timestamps of the two logs are further compared; if the difference between the timestamps of the log and the hash table is larger than a set value, storing the log into a first cache queue, and updating the timestamp of the corresponding log in a first hash table according to the timestamp of the currently input log; and storing the log in the first cache queue into an alarm log data table in the database.
In the technical scheme of the second embodiment of the application, because the information abstract is compared with the state identification bits, the similar alarm logs can be merged according to a state edge triggering mode; through the comparison of the time stamps, the redundant logs reported repeatedly within a period of time can be merged. After the redundant logs are merged, the number of the logs which are executed to be put in storage can be greatly reduced, and therefore the storage efficiency of the logs is improved.
In addition, the alarm type logs are stored in the alarm log data table in the database and are distinguished from the data tables stored in the service logs and the kernel logs, so that the readability and the operability of the whole log system are improved.
EXAMPLE III
Fig. 6 schematically shows a flowchart of a method for processing a log by a kernel log handler according to the second embodiment of the present application.
The processing method of the kernel log is similar to the alarm log, and as shown in fig. 6, the method for processing the kernel log by the kernel log processing program in the third embodiment of the present application may include the following steps:
step S601, the regular matching technology is utilized to screen out the logs with correct formats for the logs with the kernel types.
In this step, the kernel log processing program screens out the logs of the kernel type by using the regular matching technology, and the method for screening out the dirty logs may be the same as the method for screening out the logs in step S301 in fig. 3, and is not described here again.
Further, aiming at the screened kernel log with the correct format, processing is carried out according to the following steps:
step S602: the kernel log processing program compares the protocol information of the currently input log with the correct format with the protocol information of each log stored in the second hash table; if the comparison result is inconsistent, executing the following step S603 to store the currently input log into a second hash table and a second cache queue; otherwise, the following step S604 is executed;
step S603: and storing the currently input log into a second hash table and a second buffer queue.
In this step, the currently input log is stored in the second hash table, and the currently input log is also stored in the second buffer queue to be ready for storage.
Step S604: further comparing the status identification bits of the two logs with the same information abstract; if the status flag bits of the two are different, step S605 is executed to store the currently input log into the second cache queue, and the status flag bit of the corresponding log in the second hash table is updated according to the status flag bit of the currently input log; if the status flag bits are the same, step S606 is executed.
Step S605: storing the currently input log into a second cache queue, and updating a state identification bit of the corresponding log in a second hash table according to the state identification bit of the currently input log;
in this step, the currently input log is stored in a second cache queue to be put in storage, and the state identification bit of the log with the information abstract same as that of the currently input log in the second hash table is updated according to the state identification bit of the currently input log.
Step S606: further comparing the timestamps of the two logs; if the difference between the two timestamps is greater than the set value, step S607 is executed to store the log into the second cache queue, and the timestamp of the corresponding log in the second hash table is updated according to the timestamp of the currently input log; otherwise, go to step S608;
step S607: storing the currently input log into a second cache queue, and updating the time stamp of the corresponding log in a second hash table according to the time stamp of the currently input log;
step S608: the currently entered log is discarded.
In this step, the logs with the same protocol information and the same state identification bit as the currently input logs and the timestamp difference within the set value range are determined to be stored in the second hash table, and then the currently input logs are discarded as the repeated logs, so that repeated warehousing operation of a plurality of logs with the same content is avoided.
Step S609: and storing the log in the second cache queue to an alarm log data table in the database.
The log of the second cache queue can be put into a ring buffer as a snapshot to be displayed while the database insertion operation is executed, and the principle of the method is the same as that of the service log.
Therefore, by comparing the protocol information with the state identification bits, the similar kernel logs can be merged according to a state edge triggering mode; through the comparison of the time stamps, the redundant logs reported repeatedly within a period of time can be merged. After the redundant logs are merged, the number of the logs which are executed to be put in storage can be greatly reduced, and therefore the storage efficiency of the logs is improved.
Corresponding to the method for processing the kernel log by the kernel log processing program, a kernel log processing module 204 provided in the third embodiment of the present application is specifically configured to compare protocol information of a currently input log with protocol information of each log stored in a second hash table; if the comparison result is inconsistent, storing the log into a second hash table and a second cache queue; otherwise: further comparing the status identification bits of two logs with the same protocol information; if the status identification bits of the log and the hash table are different, storing the log into a second cache queue, and updating the status identification bit of the corresponding log in a second hash table according to the currently input status identification bit of the log; if the status identification bits of the two logs are the same, the timestamps of the two logs are further compared; if the difference between the timestamps of the log and the hash table is larger than a set value, storing the log into a second cache queue, and updating the timestamp of the corresponding log in a second hash table according to the timestamp of the currently input log; and storing the logs in the cache queue to a kernel log data table in the database.
In the third technical solution of the embodiment of the present application, because of the comparison between the protocol information and the state identification bits, the kernel logs of the same type can be merged according to a state edge triggering mode; through the comparison of the time stamps, the redundant logs reported repeatedly within a period of time can be merged. After the redundant logs are merged, the number of the logs which are executed to be put in storage can be greatly reduced, and therefore the storage efficiency of the logs is improved.
In addition, the logs of the kernel type are stored in the kernel log data table in the database and are distinguished from the data tables stored by the service logs and the alarm logs, so the readability and the operability of the whole log system are improved.
Example four
Fig. 7 schematically shows a hardware architecture diagram of a computer device 1000 adapted to implement the classified storage method of the log according to the fourth embodiment of the present application. In an exemplary embodiment of the present application, the computer device 1000 may be a device capable of automatically performing numerical calculation and/or information processing according to instructions set or stored in advance. For example, the server may be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server or a rack server (including an independent server or a server cluster composed of a plurality of servers), a gateway, and the like. As shown in fig. 7, the computer device 1000 includes at least, but is not limited to: the memory 1010, processor 1020, and network interface 1030 may be communicatively linked to each other via a system bus. Wherein:
the memory 1010 includes at least one type of computer-readable storage medium including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), Static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the storage 1010 may be an internal storage module of the computer device 1000, such as a hard disk or a memory of the computer device 1000. In other embodiments, the memory 1010 may be an external storage device of the computer device 1000, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), or the like, provided on the computer device 1000. Of course, the memory 1010 may also include both internal and external memory modules of the computer device 1000. In this embodiment, the memory 1010 is generally used for storing an operating system installed in the computer apparatus 1000 and various types of application software, such as program codes of a method for identifying a behavior subject of the software. In addition, the memory 1010 may also be used to temporarily store various types of data that have been output or are to be output.
Processor 1020 may be, in some embodiments, a Central Processing Unit (CPU), a controller, a microcontroller, a microprocessor, or other data Processing chip. The processor 1020 is generally configured to control the overall operation of the computer device 1000, such as performing control and processing related to data interaction or communication with the computer device 1000. In this embodiment, the processor 1020 is configured to execute program codes stored in the memory 1010 or process data.
The network interface 1030 may comprise a wireless network interface or a wired network interface, with the network interface 1030 typically being used to establish communications links between the computer device 1000 and other computer devices. For example, the network interface 1030 is used to connect the computer apparatus 1000 to an external terminal via a network, establish a data transmission channel and a communication link between the computer apparatus 1000 and the external terminal, and the like. The network may be a wireless or wired network such as an Intranet (Intranet), the Internet (Internet), a Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth (Bluetooth), or Wi-Fi.
It should be noted that FIG. 7 only shows a computer device having components 1010 and 1030, but it should be understood that not all of the shown components are required and that more or fewer components may be implemented instead.
In this embodiment, the method for identifying the behavior entity of the software stored in the memory 1010 can be further divided into one or more program modules and executed by one or more processors (in this embodiment, the processor 1020) to implement the embodiments of the present application.
EXAMPLE five
The present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method of identifying a subject of a software behavior in embodiments.
In this embodiment, the computer-readable storage medium includes a flash memory, a hard disk, a multimedia card, a card type memory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Read Only Memory (ROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Programmable Read Only Memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the computer readable storage medium may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium may be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like provided on the computer device. Of course, the computer-readable storage medium may also include both internal and external storage devices of the computer device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various types of application software installed in a computer device, for example, the program code of the method for identifying the behavior body of the software in the embodiment, and the like. Further, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.
It will be apparent to those skilled in the art that the modules or steps of the embodiments of the present application described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and alternatively, they may be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described may be performed in an order different from that described herein, or they may be separately fabricated into individual integrated circuit modules, or multiple ones of them may be fabricated into a single integrated circuit module. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application, or which are directly or indirectly applied to other related technical fields, are included in the scope of the present application.
Claims (11)
1. A method for storing logs in a classified mode is characterized by comprising the following steps:
classifying the collected logs according to key fields in the logs;
and respectively sending the classified different types of logs to corresponding log processing programs for parallel processing, and storing the logs into a database.
2. The method according to claim 1, wherein the different types of logs specifically include: at least one of a log of a service type, a log of an alarm type, and a log of a kernel type; and
the database includes at least one of the following data tables: the system comprises a service log data table for storing service type logs, an alarm log data table for storing alarm type logs and a kernel log data table for storing kernel type logs.
3. The method according to claim 2, wherein when the log is a service type log, the log is sent to a service log processing program for processing and then stored in the service log data table; the service type logs are divided into a plurality of subcategories, and the service log data tables are multiple and respectively correspond to each subcategory of the service type logs;
the method for processing the log by the service log processing program comprises the following steps: after the logs with correct formats are screened out by utilizing a regular matching technology, inserting the screened logs into a pre-constructed annular buffer queue;
reading logs one by one from the circular buffer queue;
aiming at each currently read log, identifying the sub-category of the log, and caching the log in a log storage queue corresponding to the sub-category according to the identified sub-category;
and aiming at each sub-category, when the number of the logs in the log storage queue corresponding to the sub-category reaches a set number index, inserting the logs in the log storage queue into a service log data table corresponding to the sub-category in the database in batches, and emptying the log storage queue.
4. The method according to claim 3, wherein after the regular matching technique is used to screen out the logs with the correct format, the screened logs are inserted into a pre-constructed circular buffer queue, and the method specifically comprises:
acquiring a log data block with a set size through a block analyzer, performing message format check on the acquired log data block, discarding unconventional log information in the log data block, and splitting the log data block into log message information with minimum information length;
distributing the split log message information of each row to a plurality of parallel running row resolvers through the block resolvers, and further checking the rules by the row resolvers:
and checking whether the information of a plurality of necessary fields in the input log message information of each line is in compliance or not through the line analyzer, further discarding the log message information which is not in compliance, and inserting the log message information which is in compliance into the ring buffer queue.
5. The method according to claim 4, wherein the checking, by the line parser, whether information of a number of necessary fields in each line of log message information inputted is compliant includes:
the method comprises the steps that sub-categories of log message information input within a period of time are divided through a line analyzer; and aiming at the divided log message information of each sub-category, carrying out batch rule check on the log message information of the sub-category by using an SQL statement block corresponding to the sub-category.
6. The method according to claim 2, wherein when the log is an alarm type log, the alarm type log is sent to an alarm log processing program for processing and then stored in the alarm log data table; the method for processing the log by the alarm log processing program comprises the following steps:
comparing the information abstract of the currently input log with the information abstract of each log stored in the first hash table; if the comparison result is inconsistent, storing the currently input log into a first hash table and a first cache queue; otherwise:
further comparing the status identification bits of the two logs with the same information abstract; if the status identification bits of the log and the hash table are different, storing the currently input log into a first cache queue, and updating the status identification bit of the corresponding log in a first hash table according to the status identification bit of the currently input log;
if the status identification bits of the two logs are the same, the timestamps of the two logs are further compared; if the difference between the timestamps of the log and the hash table is larger than a set value, storing the log into a first cache queue, and updating the timestamp of the corresponding log in a first hash table according to the timestamp of the currently input log;
and storing the log in the first cache queue into an alarm log data table in the database.
7. The method according to claim 2, wherein when the log is a kernel-type log, the log is sent to a kernel log processing program for processing and then stored in the kernel log data table; the method for processing the logs by the kernel log processing program comprises the following steps:
comparing the protocol information of the currently input log with the protocol information of each log stored in the second hash table; if the comparison result is inconsistent, storing the log into a second hash table and a second cache queue; otherwise:
further comparing the status identification bits of two logs with the same protocol information; if the status identification bits of the log and the hash table are different, storing the log into a second cache queue, and updating the status identification bit of the corresponding log in a second hash table according to the currently input status identification bit of the log;
if the status identification bits of the two logs are the same, the timestamps of the two logs are further compared; if the difference between the timestamps of the log and the hash table is larger than a set value, storing the log into a second cache queue, and updating the timestamp of the corresponding log in a second hash table according to the timestamp of the currently input log;
and storing the logs in the second cache queue to a kernel log data table in the database.
8. A log sorting and storing apparatus, comprising:
the log collection module is used for collecting logs and classifying the collected logs according to key fields in the logs;
and the log processing modules correspond to the classified logs of different types respectively and are used for processing the logs of the corresponding types and storing the processed logs into the database.
9. The apparatus according to claim 8, wherein the different types of logs specifically include: at least one of a log of a service type, a log of an alarm type, and a log of a kernel type; and
the database includes at least one of the following data tables: the system comprises a service log data table for storing service type logs, an alarm log data table for storing alarm type logs and a kernel log data table for storing kernel type logs.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor, when executing the computer program, is adapted to carry out the steps of the method for sorted storage of logs according to any of claims 1 to 7.
11. A computer-readable storage medium, having stored therein a computer program executable by at least one processor to cause the at least one processor to perform the steps of the method for sorted storage of logs of any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110820989.0A CN113420032A (en) | 2021-07-20 | 2021-07-20 | Classification storage method and device for logs |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110820989.0A CN113420032A (en) | 2021-07-20 | 2021-07-20 | Classification storage method and device for logs |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113420032A true CN113420032A (en) | 2021-09-21 |
Family
ID=77721516
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110820989.0A Pending CN113420032A (en) | 2021-07-20 | 2021-07-20 | Classification storage method and device for logs |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113420032A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114629786A (en) * | 2022-03-22 | 2022-06-14 | 康键信息技术（深圳）有限公司 | Log real-time analysis method, device, storage medium and system |
| CN115460214A (en) * | 2022-11-10 | 2022-12-09 | 北京天元特通科技有限公司 | Distributed network communication log storage and retrieval method and device |
| CN117112549A (en) * | 2023-10-20 | 2023-11-24 | 中科星图测控技术股份有限公司 | Big data merging method based on bloom filter |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106844143A (en) * | 2016-12-27 | 2017-06-13 | 微梦创科网络科技（中国）有限公司 | A kind of daily record duplicate removal treatment method and device |
| CN110427306A (en) * | 2019-08-12 | 2019-11-08 | 吉林吉大通信设计院股份有限公司 | A kind of big data log Intelligent routing and storage system and method |
| CN111045782A (en) * | 2019-11-20 | 2020-04-21 | 北京奇艺世纪科技有限公司 | Log processing method and device, electronic equipment and computer readable storage medium |
| WO2020253399A1 (en) * | 2019-06-21 | 2020-12-24 | 深圳前海微众银行股份有限公司 | Log classification rule generation method, device, apparatus, and readable storage medium |
| CN112612677A (en) * | 2020-12-28 | 2021-04-06 | 北京天融信网络安全技术有限公司 | Log storage method and device, electronic equipment and readable storage medium |
-
2021
- 2021-07-20 CN CN202110820989.0A patent/CN113420032A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106844143A (en) * | 2016-12-27 | 2017-06-13 | 微梦创科网络科技（中国）有限公司 | A kind of daily record duplicate removal treatment method and device |
| WO2020253399A1 (en) * | 2019-06-21 | 2020-12-24 | 深圳前海微众银行股份有限公司 | Log classification rule generation method, device, apparatus, and readable storage medium |
| CN110427306A (en) * | 2019-08-12 | 2019-11-08 | 吉林吉大通信设计院股份有限公司 | A kind of big data log Intelligent routing and storage system and method |
| CN111045782A (en) * | 2019-11-20 | 2020-04-21 | 北京奇艺世纪科技有限公司 | Log processing method and device, electronic equipment and computer readable storage medium |
| CN112612677A (en) * | 2020-12-28 | 2021-04-06 | 北京天融信网络安全技术有限公司 | Log storage method and device, electronic equipment and readable storage medium |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114629786A (en) * | 2022-03-22 | 2022-06-14 | 康键信息技术（深圳）有限公司 | Log real-time analysis method, device, storage medium and system |
| CN115460214A (en) * | 2022-11-10 | 2022-12-09 | 北京天元特通科技有限公司 | Distributed network communication log storage and retrieval method and device |
| CN115460214B (en) * | 2022-11-10 | 2023-02-07 | 北京天元特通科技有限公司 | Distributed network communication log storage and retrieval method and device |
| CN117112549A (en) * | 2023-10-20 | 2023-11-24 | 中科星图测控技术股份有限公司 | Big data merging method based on bloom filter |
| CN117112549B (en) * | 2023-10-20 | 2024-03-26 | 中科星图测控技术股份有限公司 | Big data merging method based on bloom filter |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113420032A (en) | Classification storage method and device for logs | |
| CN110928718B (en) | Abnormality processing method, system, terminal and medium based on association analysis | |
| WO2019134226A1 (en) | Log collection method, device, terminal apparatus, and storage medium | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| EP4099170B1 (en) | Method and apparatus of auditing log, electronic device, and medium | |
| US9690842B2 (en) | Analyzing frequently occurring data items | |
| CN112380473B (en) | Data acquisition and synchronization method, device, equipment and storage medium | |
| CN111881011A (en) | Log management method, platform, server and storage medium | |
| CN111538563A (en) | Event analysis method and device for Kubernetes | |
| CN111651595A (en) | Abnormal log processing method and device | |
| JP2006260056A (en) | Integrated operation management server, extraction method of message for integrative operation management, and program | |
| CN112039701A (en) | Interface call monitoring method, device, equipment and storage medium | |
| CN112015815B (en) | Data synchronization method, device and computer readable storage medium | |
| CN111611207B (en) | State data processing method and device and computer equipment | |
| CN112291214B (en) | Industrial message analysis method and system based on redis cache | |
| CN114090529A (en) | Log management method, device, system and storage medium | |
| WO2019071899A1 (en) | Electronic device, vehicle data import method and storage medium | |
| CN115328734A (en) | Cross-service log processing method and device and server | |
| WO2021129849A1 (en) | Log processing method, apparatus and device, and storage medium | |
| CN111913996B (en) | Data processing method, device, equipment and storage medium | |
| CN111444156A (en) | Fault diagnosis method based on cloud computing | |
| CN111654410B (en) | Gateway request monitoring method, device, equipment and medium | |
| CN115296976B (en) | Internet of things equipment fault detection method, device, equipment and storage medium | |
| CN116647412B (en) | Security defense method and system of Web server | |
| CN117290442A (en) | Method, system, electronic equipment and storage medium for synchronizing data among databases |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |