CN112988798A

CN112988798A - Log processing method, device, equipment and medium

Info

Publication number: CN112988798A
Application number: CN202110333018.3A
Authority: CN
Inventors: 杨松; 蒋烈
Original assignee: Chengdu Westone Information Industry Inc
Current assignee: Chengdu Westone Information Industry Inc
Priority date: 2021-03-29
Filing date: 2021-03-29
Publication date: 2021-06-18
Anticipated expiration: 2041-03-29
Also published as: CN112988798B

Abstract

The disclosure provides a log processing method, a log processing device, log processing equipment and a log processing medium, wherein the method comprises the following steps: when log data are acquired, inquiring registered counter information from a counter registry, and starting a redis database transaction; generating a counter key according to the name and the time granularity of the counter in the counter information and the time information corresponding to the time attribute field in the counter information in the log data, wherein the counter key represents a specific counter; and extracting target information corresponding to the index attribute field from the log data, updating a counter corresponding to the counter key by using the target information, and ending the redis database transaction. Therefore, log data can be counted in real time, the query response speed of log statistical information can be improved, and the flexibility of log processing is improved.

Description

Log processing method, device, equipment and medium

Technical Field

The present disclosure relates to the field of database technologies, and in particular, to a method, an apparatus, a device, and a medium for processing a log.

Background

In the security situation awareness system, hundreds of thousands of security logs are collected and stored from the security device every day, the log data are stored in the database after passing through the ETL, and the storage time limit of the data is usually required to be more than 1 year, wherein the ETL is an abbreviation of Extract-Transform-Load and is used for describing the process of extracting (Extract), converting (Transform) and loading (Load) the data from a source end to a destination end. Various statistical charts based on log data (such as 24-hour attack destination address Top5, 7-day attack event distribution trend and the like) are provided on a User Interface (UI) of the security situation awareness system, so that security analysts can analyze the current security situation and find high-risk problems.

Because the data volume of the security log is huge, Query statistics is directly performed through SQL (Structured Query Language) statements, but the Query statistics response time is too long when Query statistics is directly performed through SQL statements, so that the current common practice is to define statistical tables such as hour tables, day tables, month tables and the like for various indexes in advance, summarize and store the statistical tables in a warehouse at regular time every day, Query different tables according to different time granularities during actual statistics, and the response time is shorter than that of directly querying in an original data table. However, the way of defining the statistical tables with different time granularities requires that a database table is defined in advance, and if a new statistical index needs to be added in the operation of the system, a new statistical table and a new sql summary script need to be designed, so that the flexibility is low. And when the storage time of the original table data is very long and the data volume of the summary table is correspondingly increased, the query efficiency is also reduced if the table is not split, and meanwhile, the summary table is updated regularly, so that the requirements of real-time statistics cannot be met.

Disclosure of Invention

In view of the above, an object of the present disclosure is to provide a log processing method, apparatus, device, and medium, which can perform real-time statistics on log data, improve query response speed of log statistical information, and improve flexibility of log processing. The specific scheme is as follows:

in a first aspect, the present disclosure provides a log processing method applied to a redis database, including:

when log data are acquired, inquiring registered counter information from a counter registry, and starting a redis database transaction, wherein each counter in the counter registry is used for counting logs acquired by the redis database, the counter information comprises a counter name, a time granularity, a time attribute field and an index attribute field, and the index attribute field is field information to be recorded in the log data;

generating a counter key according to the counter name, the time granularity and the time information corresponding to the time attribute field in the log data, wherein the counter key represents a specific counter;

and extracting target information corresponding to the index attribute field from the log data, updating a counter corresponding to the counter key by using the target information, and ending the redis database transaction.

Optionally, before querying registered counter information from the counter registry when obtaining the log data, the method further includes:

reading the counter information from a preset counter definition file, wherein the counter definition file is a YAML format file;

register the counter according to the said counter information, and store the said counter information in the said counter registry;

and processing the log data, and converting the log data into data with a uniform data structure.

adding annotations in the warehousing method of the log data so as to execute the steps of querying the registered counter information from the counter registry when the log data is acquired by the redis database and the acquired log data is acquired.

Optionally, the generating a counter key according to the counter name, the time granularity, and the time information corresponding to the time attribute field in the log data includes:

determining a pre-generated counter key according to the counter name, the time granularity and the time information corresponding to the time attribute field in the log data;

judging whether the pre-generated counter key exists in the redis database or not;

and if the pre-generated counter key does not exist in the redis database, generating the counter key by using the pre-generated counter key.

Optionally, the updating, by using the target information, the counter corresponding to the counter key includes:

storing the target information into an ordered set and/or a hash in a counter corresponding to the timer key;

updating a count value corresponding to the target information in the ordered set and/or the hash;

and if the counter key does not exist in the redis database when the log data is acquired, configuring the expiration time of the counter key so as to delete the counter corresponding to the timer key when the expiration time is reached.

Optionally, the log processing method further includes:

acquiring a log statistics query instruction, wherein the log statistics query instruction comprises an index change trend query instruction and an index ranking query instruction, the index change trend query instruction is used for querying the change trend of a target index in log data in the redis database, and the index ranking query instruction is used for querying ranking information of the target index in the log data in the redis database;

and calling a preset interface to inquire the counter to be inquired corresponding to the log statistics inquiry instruction, and returning an inquiry result corresponding to the log statistics inquiry instruction.

Optionally, the invoking a preset interface to query the counter to be queried corresponding to the log statistics query instruction, and returning a query result corresponding to the log statistics query instruction includes:

and calling a preset interface to inquire a counter to be inquired corresponding to the index change trend inquiry instruction, and returning an inquiry result corresponding to the index change trend inquiry instruction, wherein the index change trend inquiry instruction comprises the name of the counter to be inquired, an index to be inquired, the granularity of time to be inquired, the range of time to be inquired and the inquiry starting time.

Optionally, the invoking a preset interface to query the counter to be queried corresponding to the index change trend query instruction, and returning a query result corresponding to the index change trend query instruction includes:

determining a first time to be queried key based on the time to be queried granularity, the time range to be queried and the query starting time, wherein the first time to be queried key represents the time to be queried;

determining a first to-be-queried counter key based on the to-be-queried counter name and the first to-be-queried time key;

and inquiring the count value corresponding to the index to be inquired in the first counter to be inquired corresponding to the first counter to be inquired key, and returning the count value corresponding to the index to be inquired.

and calling a preset interface to inquire a counter to be inquired corresponding to the index ranking inquiry instruction, and returning an inquiry result corresponding to the index ranking inquiry instruction, wherein the index ranking inquiry instruction comprises the name of the counter to be inquired, the granularity of time to be inquired, the range of time to be inquired, the inquiry starting time and the ranking range.

Optionally, the invoking a preset interface to query the counter to be queried corresponding to the index ranking query instruction, and returning a query result corresponding to the index ranking query instruction includes:

determining a second time key to be queried based on the time granularity to be queried, the time range to be queried and the query starting time, wherein the second time key to be queried represents the time to be queried;

determining a second counter key to be queried based on the counter name to be queried and the second time key to be queried;

merging the second counters to be queried corresponding to the second counters to be queried key to generate a merged counter;

and returning the index information of which the counted value ranking meets the ranking range in the merged counter as a query result corresponding to the index ranking query instruction.

In a second aspect, the present disclosure provides a log processing apparatus applied to a redis database, where the redis database includes a counter recording module, and the counter recording module includes:

the query unit is configured to query registered counter information from a counter registry and start a redis database transaction when log data is acquired, where each counter in the counter registry is used to count logs acquired by the redis database, the counter information includes a counter name, a time granularity, a time attribute field, and an index attribute field, and the index attribute field is field information to be recorded in the log data;

a Key generating unit, configured to generate a counter Key according to the counter name, the time granularity, and time information corresponding to the time attribute field in the log data, where the counter Key represents a specific counter;

and the counting unit is used for extracting target information corresponding to the index attribute field from the log data, updating a counter corresponding to the counter key by using the target information, and ending the redis database transaction.

In a third aspect, the present disclosure provides an electronic device, comprising:

a memory and a processor;

wherein the memory is used for storing a computer program;

the processor is configured to execute the computer program to implement the log processing method disclosed in the foregoing.

In a fourth aspect, the present disclosure proposes a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the log processing method disclosed in the foregoing.

It can be seen that the present disclosure is applied to a redis database, and when log data is obtained, registered counter information is queried from a counter registry, and a transaction of the redis database is started, wherein each counter in the counter registry is used for counting logs obtained by the redis database, the counter information includes a counter name, a time granularity, a time attribute field, and an index attribute field, the index attribute field is field information to be recorded in the log data, and then a counter key is generated according to the counter name, the time granularity, and time information corresponding to the time attribute field in the log data, wherein the counter key represents a specific counter, then target information corresponding to the index attribute field is extracted from the log data, and the counter corresponding to the counter key is updated by using the target information, ending the redis database transaction. Therefore, according to the method and the device, when the database acquires the log data, the log data are subjected to real-time statistical operation, the real-time statistical data can be obtained, the counter is updated when the log data are stored in the database, and resources consumed in later-period summarization are avoided. In the process of counting the log data, a counter key can be generated based on the registered counter information and the time information in the log data, and the related information of the log data is updated to a counter corresponding to the generated counter key, and the counter can be dynamically added based on the registered counter information and the time information in the log data, so that the flexibility of log processing is improved. And the counter comprises the result obtained by counting the log data stored in the database, so that when the calculator queries, additional statistical operation cannot be triggered, the query efficiency cannot be reduced along with the increase of the original data, and the query response speed of the log statistical information is improved. In addition, due to the characteristic of high-speed reading and writing of the redis database, the updating and inquiring speeds of the counter in the redis database are very high. Due to the transaction and pipeline functions of the redis database, the updating operation which can be finished only by interacting with the redis database for hundreds of times in the original updating process of the counter can be reduced to several times, and the efficiency of the counter is improved.

Drawings

In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present disclosure, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

Fig. 1 is a flowchart of a log processing method proposed by the present disclosure;

fig. 2 is a storage structure of a one-dimensional counter whose index attribute field is an attack target IP according to the present disclosure;

fig. 3 is a storage structure of a two-dimensional counter with an index attribute field as an attack target IP and an attack type according to the present disclosure;

fig. 4 is a data structure of log data according to the present disclosure;

FIG. 5 is a flowchart of a specific log processing proposed by the present disclosure;

fig. 6 is a flowchart of a specific log processing method proposed in the present disclosure;

FIG. 7 is a flow chart of a log statistics query proposed by the present disclosure;

fig. 8 is a flow chart of log statistics query proposed by the present disclosure;

fig. 9 is a flow chart of log statistics query proposed by the present disclosure;

FIG. 10 is a flow chart of a log statistics query proposed by the present disclosure;

fig. 11 is a schematic structural diagram of a log processing apparatus according to the present disclosure;

fig. 12 is a schematic structural diagram of an electronic device according to the present disclosure.

Detailed Description

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.

Referring to fig. 1, an embodiment of the present disclosure discloses a log processing method applied to a redis database, where the method includes:

step S11: when log data are acquired, registered counter information is inquired from a counter registry, and a redis database transaction is started, wherein each counter in the counter registry is used for counting logs acquired by the redis database, the counter information comprises a counter name, time granularity, a time attribute field and an index attribute field, and the index attribute field is field information to be recorded in the log data.

In a specific implementation process, when log data is acquired, registered counter information needs to be inquired from a counter registry first, and a redis database transaction is started, wherein the counter registry comprises information of registered counters, each counter is used for counting logs acquired by the redis database, the counter information comprises a counter name, time granularity, a time attribute field and an index attribute field, and the index attribute field is field information to be recorded in the log data. For example, the attribute field may be an attack target IP (Internet Protocol).

The Redis database is a key-value type NoSQL memory database, written by ANSI C, complying with BSD (Berkeley Software Distribution) protocol, and providing API (Application Programming Interface) of multiple languages. The biggest characteristic of the Redis database is that all data can be stored in a memory, so the read-write speed performance is very good. The Redis database also supports the persistence of data in memory to the hard disk in the form of snapshots and logs, so that data is not lost even when abnormal conditions such as power failure, machine failure, etc. occur. Redis databases have rich data types, such as strings, hashes, lists, collections, ordered collections, and the like. The Redis database supports transactions, the operation is atomicity, the expiration time is set according to keys, data can be automatically deleted after expiration, and due to the rich data types of the Redis database, the application scenarios are very many. The counter will use an ordered set, hash both data structures, and the command and key auto-expiration functions associated with both data structures.

The storage structure of the counter comprises an ordered set, or an ordered set plus a hash. The index attribute field in the counter information may be from one dimension to N dimensions, for example, statistics is performed only according to the attack target IP, which is one-dimensional statistics. And carrying out statistics according to the attack target IP, and simultaneously carrying out statistics on the attack type distribution under each attack target IP, wherein the statistics is two-dimensional statistics. In consideration of the intuitiveness of the graph presentation, the statistical data in the actual scene are all one-dimensional or two-dimensional, so the counter in the disclosure mainly designs a storage structure for the two statistical data, wherein the index attribute field is one-dimensional counting, the counter stores data by using an ordered set of a redis database, and the index attribute field is two-dimensional counting, and the counter stores data by associating hashes on the basis of the ordered set.

The ordered set (zset) of the Redis database may hold a series of non-repeating elements, each containing two attributes: members (member) and scores (score), and provides processing commands (increase or decrease) for scores, and commands (which can be sorted quickly) to get or scan members and scores in order according to the size of scores. In the one-dimensional counter, one element of the ordered set represents a classification count, wherein memer stores a classification name, and score stores a count value, for example, in the counter in which the index attribute field is the attack target IP, memer stores a specific attack target IP address, and score stores the number of times the attack target IP address occurs. For example, as shown in fig. 2, a storage structure of a one-dimensional counter in which the index attribute field is an attack target IP is shown. The index attribute field is only an attack target IP, so that the index attribute field is a one-dimensional counter and is counted by adopting an ordered set (ZSET), wherein memer stores a specific attack target IP address, and score stores the occurrence frequency of the attack target IP address. For example, when the attack target IP is 10.1.2.3, the number of occurrences is 12. When the attack target IP is 10.1.2.5, the number of occurrences is 18. When the attack target IP is 10.1.3.13, the number of occurrences is 78. When the attack target IP is 10.2.5.1, the number of occurrences is 115, and the like.

Hashes of Redis databases allow users to store multiple key-value pairs (subKey-values) into a key, making hashes well suited for storing some related data together. In a two-dimensional counter, data of a first dimension is still stored by using an ordered set, data of a second dimension is associated by using a hash, wherein a hashed key stores a classification name, a subKey stores a subcategory, and a value stores a count value, and the hashed key stores the classification name which can be in a form of adding the content of a member in the ordered set to the name of the counter. For example, in counters of an attack target IP and an attack type, a key storage comprises a specific IP address, for example, attack, type, count, 10.1.2.3, wherein the attack, type, count, counter name, 10.1.2.3 are specific attack target IP addresses, a subKey stores the attack type, and a value stores the number of times the attack type occurs. As shown in fig. 3. And the index attribute field is a storage structure of a two-dimensional counter of an attack target IP and an attack type. On the basis of fig. 2, hashes are associated to store information related to attack types, for example, a key corresponding to a hash is attack: type: count:10.1.2.3, a subKey stored in a hash includes XSS injection, a corresponding number of times is 3, SQL injection, a corresponding number of times is 5, and the like. The key corresponding to the hash is attach, type, count, 10.2.5.1, the subKey stored in the hash comprises XSS injection with 18 times, SQL injection with 36 times, and the like.

When the log data is acquired, before querying the registered counter information from the counter registry, the method further includes: reading the counter information from a preset counter definition file, wherein the counter definition file is a YAML (YAML Ain't Markup Language) format file; register the counter according to the said counter information, and store the said counter information in the said counter registry; and processing the log data, and converting the log data into data with a uniform data structure.

That is, it is necessary to register a counter according to the counter information included in a preset counter definition file, and store the counter information in the counter registry. The counter information comprises a counter name, time granularity, a time attribute field and an index attribute field, and the index attribute field is field information to be recorded in the log data.

In particular, the counter name (name) needs to be globally unique so that the counter is looked up by this name when the code executes. The index attribute field (subcategory field) is one (one-dimensional statistics) or two (two-dimensional statistics) attributes selected from log data as index attributes to be counted, such as an attack target IP, an attack type, and the like. The time granularity supported by the counter (timeUnit) is: the counter has a default storage life of each counter, wherein the counter is automatically deleted once the counter expires, for example, the default storage life of a minute counter is 60 minutes, the default storage life of an hour counter is 24 hours, and the default storage life of a day counter is 31 days, and the storage life can be customized when the counter is set. The time attribute field (timeField) also needs to read the time information in the log data. In addition, the counter information may further include a counter alias (alias), and the counter may be assigned with an alias with high readability for query presentation. The format of the counter definition file may be as follows:

counters:

-name:attack:destip:type:count

alias ` attack IP type count `'

categoryField:destIp

subCategoryField:eventName

timeUnit:h(48)|d|M(6)|A

timeField:eventGenerateTime

-name:event:type:count

Alias: 'event type count'

categoryField:opObjectType

timeUnit:h|d|w|M

timeField:eventGenerateTime

h (48) indicates that the shelf life when the time granularity is hour is changed from the default 24 hours to 48 hours, and M (6) indicates that the shelf life when the time granularity is month is changed from the default to 6 months.

Reading the counter information in the preset counter definition file, registering the counter according to the counter information, and storing the counter information into the counter registration information table after registration so as to be used in the subsequent updating of the counter and the query of the counter. Therefore, counters can be dynamically added or deleted, and flexibility is good.

In order to facilitate log statistics and improve efficiency in log processing, the log data needs to be processed and converted into data with a uniform data structure. Therefore, the information of the preset field is conveniently extracted from the log data, otherwise, the information of the corresponding field cannot be found due to non-uniform data structure. Wherein, the data structure of the log data can be unified as shown in fig. 4.

When the log data is acquired, the registered counter information needs to be queried from the counter registry first, so as to determine which types of counters are included in the redis database, and which counters need to be updated by using the log data. After determining which types of counters are included in the redis database, the redis database can be opened to perform statistics on the log data.

Step S12: and generating a counter key according to the counter name, the time granularity and the time information corresponding to the time attribute field in the log data, wherein the counter key represents a specific counter.

It can be understood that a counter key is further required to be generated according to the counter name, the time granularity and the time information corresponding to the time attribute field in the log data, where the counter key represents a specific counter.

Specifically, the same counter name pair corresponds to different time granularities, so the same counter name may correspond to multiple counter keys, for example, the counter name is attack: destip: type: count, the time granularity is day and month, and if the counter is registered from 20

days

12 and 20 months 2020, the counter key corresponding to the counter name may include attack: destip: type: count: 20201221, attack: destip: type: count: 20201222, attack: destip: type: count: 20201223, etc., and may also include an attack: destination: type: count: 202012, attack: destip: type: count: 202101. each counter has its own counter. Therefore, a counter key needs to be generated according to the counter name, the time granularity and the time information corresponding to the time attribute field in the log data, so as to determine a specific counter needing to be updated.

Specifically, generating a counter key according to the counter name, the time granularity, and the time information corresponding to the time attribute field in the log data includes: determining a pre-generated counter key according to the counter name, the time granularity and the time information corresponding to the time attribute field in the log data; judging whether the pre-generated counter key exists in the redis database or not; and if the pre-generated counter key does not exist in the redis database, generating the counter key by using the pre-generated counter key. The number of pre-generation counters key is greater than or equal to 1.

That is, determining a pre-generated counter key to be generated according to the counter name, the time granularity, and the time information corresponding to the time attribute field in the log data, actually determining whether a counter key corresponding to the pre-generated counter key needs to be generated, and determining whether the pre-generated counter key already exists in the redis database, if not, generating the counter key by using the pre-generated counter key, and if so, indicating that a counter corresponding to the pre-generated counter key already exists and does not need to be newly generated.

For example, if the name of the counter is event: type: count, the supported time granularity is hour, day, week, month, the time information in the log data is 2018-10-17T11:13:52.000+0800, there are 4 pre-generated counters key, representing the counters of 4 time granularities, which are as follows:

event:type:count:2018101711

event:type:count:20181017

event:type:count:2018-45

event:type:count:201810

then, it needs to judge whether each pre-generated counter key already exists in the redis database, if yes, the corresponding counter key does not need to be generated, and if not, a new counter key needs to be generated. For example, if the event type count 2018101711 does not exist in the redis database, then a counter key is generated: event type account 2018101711.

Step S13: and extracting target information corresponding to the index attribute field from the log data, updating a counter corresponding to the counter key by using the target information, and ending the redis database transaction.

It can be understood that target information corresponding to the index attribute field needs to be extracted from the log data, and the counter corresponding to the counter key is updated by using the target information, so as to end the redis database transaction, where the counter corresponding to the counter key includes an ordered set, or an ordered set plus a hash.

Updating the counter corresponding to the counter key by using the target information, wherein the updating comprises the following steps: storing the target information into an ordered set and/or a hash in a counter corresponding to the timer key; updating a count value corresponding to the target information in the ordered set and/or the hash; and if the counter key does not exist in the redis database when the log data is acquired, configuring the expiration time of the counter key so as to delete the counter corresponding to the timer key when the expiration time is reached.

That is, the target information is stored into the member in the ordered set and/or the subKey in the hash in the counter corresponding to the timer key, and then the corresponding count value is accumulated. If the target information is stored in the member in the ordered set in the counter corresponding to the timer key and/or the subKey in the hash, the corresponding count value can be directly updated.

For example, if the index attribute field is an attack target IP, and the counter key is attack, destination, type, count, 2018101711, extracting an attack target IP address in the log data, and then updating a counter corresponding to attack, destination, type, count, 2018101711 by using the extracted target IP address. The index attribute field only comprises an attack target IP (Internet protocol), namely a one-dimensional counter, so that the counter corresponding to attack: type: count:2018101711 is an ordered set, if the number in the ordered set does not have the extracted attack target IP, the extracted attack target IP is stored in the number in the ordered set, and the corresponding count value is updated, and if the number in the ordered set has the extracted attack target IP, the corresponding count value is directly updated.

Specifically, an increment command of a redis database is called, so that N can be directly added to a corresponding counter, where N is a count value to be accumulated. If the counter has no value before, the default is set to be N, if the counter is a two-dimensional counter, then an increment command is called, and the updating operation of the two-dimensional counter is completed.

In addition, after updating the counter corresponding to the counter key, the method further includes: and if the counter key does not exist in the redis database when the log data is acquired, configuring the expiration time of the counter key so as to delete the counter corresponding to the timer key when the expiration time is reached. That is, if the counter key is a newly generated counter key, the expiration time of the counter key also needs to be configured, so that when the expiration time is reached, the counter corresponding to the timer key is deleted. If the counter key is already stored in the redis database when the log data is acquired, the expiration time of the counter key is already set. Specifically, an expire (redisKey, ttl) command of the redis database can be called, the expiration time of the counter can be set, and the counter can be automatically deleted after the expiration time, so that the memory space is saved. The counters with different time granularities have different storage time ranges, and the counters can be automatically deleted after the counters are expired, so that manual intervention is not needed, and the memory space is saved.

If the log data comprises a plurality of pieces of log data, the accumulated value can be obtained through internal circulation and then written into a counter in a redis database, so that the efficiency is improved.

When the redis affairs are finished, the operation on the redis database is actually executed before, so that the consistency of operation results is ensured, the efficiency is improved, and in the actual test, the recording operation of the counter can be finished within tens of milliseconds under the condition of one thousand pieces of log data.

Referring to fig. 5, a flow chart of log processing is shown. Firstly, obtaining log data, then inquiring out counter information from a counter registry, starting a redis database transaction, and judging whether an unprocessed timer exists, namely judging whether a registered counter in the redis database needs to be updated according to the log data. If so, generating a counter key according to the counter name, the time granularity and the time information corresponding to the time attribute field in the log data, extracting target information corresponding to the index attribute field from the log data as a member in an ordered set and/or a subKey in a hash, updating a count value corresponding to the target information in the ordered set and/or the hash, setting the expiration time of the counter key when the counter key is newly generated, judging whether the counter key corresponding to the log information is processed, if so, judging whether an unprocessed counter exists, and if not, ending the redis database transaction.

In a specific implementation process, before querying registered counter information from the counter registry when log data is acquired, the method further includes: adding annotations in the warehousing method of the log data so as to execute the steps of querying the registered counter information from the counter registry when the log data is acquired by the redis database and the acquired log data is acquired. That is, an @ addCounter comment may be added to the logging method of the log data, so that when the redis database acquires the log data, the step of querying the registered counter information from the counter registry when acquiring the log data is executed. The counter is easy to use, the main work is to define a counter definition file, and then corresponding annotations are added to a method for updating and warehousing log data.

Referring to fig. 6, an embodiment of the present disclosure discloses a specific log processing method applied to a redis database, where the method includes:

step S21: when log data are acquired, registered counter information is inquired from a counter registry, and a redis database transaction is started, wherein each counter in the counter registry is used for counting logs acquired by the redis database, the counter information comprises a counter name, time granularity, a time attribute field and an index attribute field, and the index attribute field is field information to be recorded in the log data.

Step S22: and generating a counter key according to the counter name, the time granularity and the time information corresponding to the time attribute field in the log data, wherein the counter key represents a specific counter.

Step S23: and extracting target information corresponding to the index attribute field from the log data, updating a counter corresponding to the counter key by using the target information, and ending the redis database transaction.

The specific implementation process of step S21 to step S23 may refer to the content disclosed in the foregoing embodiments, and will not be described herein again.

Step S24: obtaining a log statistics query instruction, wherein the log statistics query instruction comprises an index change trend query instruction and an index ranking query instruction, the index change trend query instruction is used for querying a change trend of a target index in log data in the redis database, and the index ranking query instruction is used for querying ranking information of the target index in the log data in the redis database.

The log processing method further comprises the following steps: and acquiring a log statistics query instruction so as to query the log information counted in the counter, wherein the log statistics query instruction comprises an index change trend query instruction and an index ranking query instruction, and the index change trend query instruction is used for querying a change trend of a target index in the log data in the redis database, for example, an attack trend of the latest 7 days. The index ranking query instruction is used for querying ranking information of a target index in log data in the redis database, for example, a target IP Top5 attacked in the last 24 hours.

Step S25: and calling a preset interface to inquire the counter to be inquired corresponding to the log statistics inquiry instruction, and returning an inquiry result corresponding to the log statistics inquiry instruction.

It can be understood that after the log statistics query instruction is obtained, a preset interface is called to query the counter to be queried corresponding to the log statistics query instruction, and a corresponding query result is returned.

Specifically, invoking a preset interface to query a counter to be queried corresponding to the log statistics query instruction, and returning a query result corresponding to the log statistics query instruction includes: and calling a preset interface to inquire a counter to be inquired corresponding to the index change trend inquiry instruction, and returning an inquiry result corresponding to the index change trend inquiry instruction, wherein the index change trend inquiry instruction comprises the name of the counter to be inquired, an index to be inquired, the granularity of time to be inquired, the range of time to be inquired and the inquiry starting time.

Specifically, invoking a preset interface to query a counter to be queried corresponding to the log statistics query instruction, and returning a query result corresponding to the log statistics query instruction includes: and calling a preset interface to inquire a counter to be inquired corresponding to the index ranking inquiry instruction, and returning an inquiry result corresponding to the index ranking inquiry instruction, wherein the index ranking inquiry instruction comprises the name of the counter to be inquired, the granularity of time to be inquired, the range of time to be inquired, the inquiry starting time and the ranking range.

Referring to fig. 7, the invoking a preset interface to query the counter to be queried corresponding to the index change trend query instruction, and returning a query result corresponding to the index change trend query instruction includes:

step S31, determining a first time to be queried key based on the time to be queried granularity, the time range to be queried and the query start time, wherein the first time to be queried key represents the time to be queried.

Step S32, determining a first to-be-queried counter key based on the to-be-queried counter name and the first to-be-queried time key.

And step S33, inquiring the count value corresponding to the index to be inquired in the first counter to be inquired corresponding to the first counter to be inquired key, and returning the count value corresponding to the index to be inquired.

When the log statistics query instruction is the index variation trend query instruction, the default interface call may be List < Map < String, Long > > timeUnit counter (String counter, String category, String timeUnit, int timeRange, Date status Date), where counter is a name of a counter to be queried, category is an index to be queried, timeUnit is a granularity of time to be queried, timeRange is a time range to be queried, and startDate is a query start time. For example, the trend of inquiring and returning 10.1.2.3 that this IP address was attacked in the last 24 hours may be: timeUnitcounter ("ack: destip: count", "10.1.2.3", "hour", 24, now ()).

First, a first time key to be queried is determined based on the time granularity to be queried, the time range to be queried and the query start time. For example, when the query start time is 2018, 10, 17, 11, the time range to be queried is the past 24 hours, and the granularity of the time to be queried is small, the first time to be queried key is as follows:

2018101711

2018101710

2018101709

2018101708

……

2018101612

and then determining a first to-be-queried counter key based on the to-be-queried counter name and the first to-be-queried time key. For example, if the to-be-queried counter is named as "ack": destip: count ", the first to-be-queried counter key is as follows:

attack:destip:count:2018101711

attack:destip:count:2018101710

attack:destip:count:2018101709

attack:destip:count:2018101708

……

attack:destip:count:2018101612

then, a count value score corresponding to the index to be queried in the first counter to be queried corresponding to the first counter to be queried key can be queried, and the count value score corresponding to the index to be queried is returned. The score value is a counting value, and if a two-dimensional index exists, a specific counting value is inquired through a counter key of the two-dimensional index, wherein the attribute is the count:10.1.2.3: 2018101711.

Referring to fig. 8, a flow chart of log statistics query is shown. First, a first time key to be queried is determined based on the time granularity to be queried, the time range to be queried and the query start time. And then determining a first to-be-queried counter key based on the to-be-queried counter name and the first to-be-queried time key. And judging whether an unprocessed first counter to be inquired key exists or not, and if so, inquiring the count value corresponding to the index to be inquired in the first counter to be inquired corresponding to the first counter to be inquired key. And if not, combining and returning the count values corresponding to the indexes to be inquired.

Referring to fig. 9, the invoking a preset interface to query the counter to be queried corresponding to the index ranking query instruction and returning a query result corresponding to the index ranking query instruction includes:

step S41: and determining a second time key to be queried based on the time granularity to be queried, the time range to be queried and the query starting time, wherein the second time key to be queried represents the time to be queried.

Step S42: and determining a second counter key to be queried based on the counter name to be queried and the second time key to be queried.

Step S43: and merging the second counters to be inquired corresponding to the second counters to be inquired key to generate a merged counter.

Step S44: and returning the index information of which the counted value ranking meets the ranking range in the merged counter as a query result corresponding to the index ranking query instruction.

When the log statistics query instruction is the index variation trend query instruction, the preset interface call may be List < Map < String, Long > > aggregater (String counter, String timeUnit, int timeRange, Date status Date, int topN), where the counter to be queried name, timeUnit to be queried time granularity, timeRange to be queried time range, startDate query start time, and topN represent a ranking range. For example, the attacked target IP address 5 bits ranked top of the last 7 days is queried and returned. It can be: aggregate counter ("ack: destip: count", "day", 7, now (), 5).

First, a first time key to be queried is determined based on the time granularity to be queried, the time range to be queried and the query start time. And then determining a second counter key to be queried based on the counter name to be queried and the second time key to be queried. And then merging the second counters to be inquired corresponding to the second counters to be inquired key to generate a merged counter. Specifically, a zunionStore (keySet, tmpKey) command of the redis database may be called to perform union aggregation on the second counters to be queried corresponding to the second counters to be queried, score values of the same members in the second counters to be queried may be accumulated, and the members are sorted according to new scores in the generated temporary ordered set in the merged counters.

And then returning the index information of which the row of the counting values in the merged counter meets the row range as a query result corresponding to the index row query instruction. The index information can be an attack target IP, an attack type and the like. Specifically, a zrevRange (tmpKey,0, topN) command of redis may be called to return an element of score ranking topN in the ordered set in the timer after merging, and if there is a two-dimensional index, for each element of topN, the two-dimensional index is looked up in each two-dimensional counter key for merging.

Referring to fig. 10, a flow chart of log statistics query is shown. First, a first time key to be queried is determined based on the time granularity to be queried, the time range to be queried and the query start time. And then determining a second counter key to be queried based on the counter name to be queried and the second time key to be queried. And then merging the second counters to be inquired corresponding to the second counters to be inquired key to generate a merged counter. The elements of score rank topN are taken from the ordered set in the timer after merging and the formatting result is returned.

Referring to fig. 11, an embodiment of the present disclosure discloses a log processing apparatus, which is applied to a redis database, where the redis database includes a counter recording module 11, and the counter recording module 11 includes:

the query unit 111 is configured to, when log data is obtained, query registered counter information from a counter registry, and open a redis database transaction, where each counter in the counter registry is used to perform statistics on a log obtained by the redis database, the counter information includes a counter name, a time granularity, a time attribute field, and an index attribute field, and the index attribute field is field information to be recorded in the log data;

a Key generating unit 112, configured to generate a counter Key according to the counter name, the time granularity, and time information corresponding to the time attribute field in the log data, where the counter Key represents a specific counter;

and the counting unit 113 is configured to extract target information corresponding to the index attribute field from the log data, update a counter corresponding to the counter key by using the target information, and end the redis database transaction, where the counter corresponding to the counter key includes an ordered set and/or a hash.

In some specific implementation procedures, the log processing apparatus further includes:

a counter registration module 12, configured to read the counter information from a preset counter definition file, where the counter definition file is a file in YAML format; register the counter according to the said counter information, and store the said counter information in the said counter registry;

and the data processing module is used for processing the log data and converting the log data into data with a uniform data structure.

an annotation adding module 13, configured to add an annotation to the logging method of the log data, so that when the redis database acquires the log data, and when the acquired log data is acquired, a step of starting to query registered counter information from a counter registry is performed.

In some specific implementations, the Key generating unit 112 is configured to:

In some specific implementations, the counting unit 113 is configured to:

the counter query module 14 is configured to obtain a log statistics query instruction, where the log statistics query instruction includes an index change trend query instruction and an index ranking query instruction, the index change trend query instruction is used to query a change trend of a target index in log data in the redis database, and the index ranking query instruction is used to query ranking information of the target index in the log data in the redis database; and calling a preset interface to inquire the counter to be inquired corresponding to the log statistics inquiry instruction, and returning an inquiry result corresponding to the log statistics inquiry instruction.

In some specific implementations, the counter query module 14 is configured to:

Further, an embodiment of the present disclosure also discloses a computer readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the log processing method disclosed in any of the foregoing embodiments.

For the specific process of the log processing method, reference may be made to corresponding contents disclosed in the foregoing embodiments, and details are not repeated here.

FIG. 12 is a block diagram illustrating one type of electronic device 20 according to an example embodiment. The electronic device 20 comprises a processor 21 and a memory 22. The electronic device 20 may also include one or more of a multimedia component 23, an input/output (I/O) interface 24, and a communications component 25.

The processor 21 is configured to control the overall operation of the electronic device 20, so as to complete all or part of the steps in the multi-user name and password authentication method. The memory 22 is used to store various types of data to support operation at the electronic device 20, such as instructions for any application or method operating on the electronic device 20, and application-related data, such as contact data, messaging, pictures, audio, video, and so forth. The Memory 22 may be implemented by any type of volatile or non-volatile Memory device or combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic Memory, flash Memory, magnetic disk or optical disk. The multimedia components 23 may include a screen and an audio component. Wherein the screen may be, for example, a touch screen and the audio component is used for outputting and/or inputting audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may further be stored in the memory 22 or transmitted via the communication component 25. The audio assembly also includes at least one speaker for outputting audio signals. The I/O interface 24 provides an interface between the processor 21 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 25 is used for wired or wireless communication between the electronic device 20 and other devices. Wireless Communication, such as Wi-Fi, bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so that the corresponding Communication component 25 may include: Wi-Fi module, bluetooth module, NFC module.

In an exemplary embodiment, the electronic Device 20 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components for executing the above-mentioned log Processing method.

The preferred embodiments of the present disclosure are described in detail with reference to the accompanying drawings, however, the present disclosure is not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solution of the present disclosure within the technical idea of the present disclosure, and these simple modifications all belong to the protection scope of the present disclosure.

It should be noted that, in the foregoing embodiments, various features described in the above embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, various combinations that are possible in the present disclosure are not described again.

In addition, any combination of various embodiments of the present disclosure may be made, and the same should be considered as the disclosure of the present disclosure, as long as it does not depart from the spirit of the present disclosure.

Finally, it is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of other elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

Claims

1. A log processing method is applied to a redis database and comprises the following steps:

2. The log processing method according to claim 1, wherein before querying the registered counter information from the counter registry when the log data is acquired, the method further comprises:

3. The log processing method according to claim 1, wherein before querying the registered counter information from the counter registry when the log data is acquired, the method further comprises:

4. The log processing method according to claim 1, wherein the generating a counter key according to the counter name, the time granularity, and the time information corresponding to the time attribute field in the log data comprises:

5. The log processing method according to claim 1, wherein the updating the counter corresponding to the counter key by using the target information includes:

6. The log processing method according to any one of claims 1 to 5, further comprising:

7. The log processing method according to claim 6, wherein the invoking a preset interface queries a counter to be queried corresponding to the log statistics query instruction and returns a query result corresponding to the log statistics query instruction, and the method comprises:

8. The log processing method according to claim 7, wherein the invoking a preset interface queries a counter to be queried corresponding to the index change trend query instruction, and returns a query result corresponding to the index change trend query instruction, and includes:

9. The log processing method according to claim 6, wherein the invoking a preset interface queries a counter to be queried corresponding to the log statistics query instruction and returns a query result corresponding to the log statistics query instruction, and the method comprises:

10. The log processing method according to claim 9, wherein the invoking a preset interface queries a counter to be queried corresponding to the index ranking query instruction, and returns a query result corresponding to the index ranking query instruction, and includes:

11. A log processing device is applied to a redis database, wherein the redis database includes a counter recording module, and the counter recording module includes:

12. An electronic device, comprising:

a memory and a processor;

wherein the memory is used for storing a computer program;

the processor is configured to execute the computer program to implement the log processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the log processing method according to any one of claims 1 to 10.