WO2020253399A1 - 日志分类规则的生成方法、装置、设备及可读存储介质 - Google Patents

日志分类规则的生成方法、装置、设备及可读存储介质 Download PDF

Info

Publication number
WO2020253399A1
WO2020253399A1 PCT/CN2020/087835 CN2020087835W WO2020253399A1 WO 2020253399 A1 WO2020253399 A1 WO 2020253399A1 CN 2020087835 W CN2020087835 W CN 2020087835W WO 2020253399 A1 WO2020253399 A1 WO 2020253399A1
Authority
WO
WIPO (PCT)
Prior art keywords
log
array
information template
category
feature
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/087835
Other languages
English (en)
French (fr)
Inventor
李悦
曾可
卢道和
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
WeBank Co Ltd
Original Assignee
WeBank Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by WeBank Co Ltd filed Critical WeBank Co Ltd
Publication of WO2020253399A1 publication Critical patent/WO2020253399A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Definitions

  • This application relates to the field of Fintech data processing technology, and in particular to a method, device, device and readable storage medium for generating log classification rules.
  • text records are generated in network devices such as switches, routers, and firewalls.
  • the text records are logs.
  • various network equipment manufacturers record events encountered during the operation of network equipment such as switches, routers, and firewalls as logs.
  • each line of the log describes the latest state of an object at a certain point in time.
  • the sentence format describing each type of event is inconsistent, and the sentence format of similar events on different device models may also be inconsistent, the log is unstructured data with irregular structure.
  • Unstructured data is data with irregular or incomplete data structure, no predefined data model, and inconvenient data represented by two-dimensional logical tables of a database, such as various types of text.
  • the first one is to manually analyze the log data of the network equipment according to the network equipment manufacturer and model, and then define the regular expression rules for each type of event, which is defined by a preset program.
  • the regular expression rules of the log are classified.
  • the second is to use the text similarity algorithm to calculate the similarity between each line of log samples. The similarity is greater than the set value for the same type of samples.
  • the same type of samples are used to generate the regular expression rules required for classification, and finally according to the regular expression rules Categorize the logs.
  • the first log classification method requires manual analysis of logs and the definition of regular expressions required for classification, which requires a lot of work, high cost of log classification, and low efficiency of regular expression rule generation;
  • the second log classification method uses regular expressions from log samples , And classification depends on the set value of similarity between log samples. This method can only indirectly intervene the regular expression rules required for classification through the set value, and the process of generating regular expression rules is cumbersome.
  • the main purpose of this application is to provide a method, device, device, and readable storage medium for generating log classification rules, aiming to solve the existing technical problems of cumbersome and inefficient regular expression rules required for generating log classification.
  • the present application provides a method for generating log classification rules.
  • the method for generating log classification rules includes the following steps:
  • the log category corresponding to the log to be classified is determined, and the log set corresponding to each log category is determined;
  • the feature array corresponding to the log set is determined according to the first information template, and the log information is generated based on the feature array and the category data corresponding to the log category Classification rules.
  • this application also provides a log classification rule generation device, the log classification rule generation device includes:
  • the determining module is configured to determine the log category corresponding to the log to be classified when the log to be classified is obtained, and determine the log set corresponding to each log category;
  • a reading module configured to read a log array in the log set and determine it as a target array
  • a calculation module for calculating the first difference between the target array and other log arrays in the log set
  • the determining module is further configured to determine a first information template corresponding to the log set according to the first difference;
  • the detection module is used to detect whether there is only one kind of the first information template
  • the determining module is further configured to determine the feature array corresponding to the log set according to the first information template if only one type of the first information template is detected;
  • the generating module is configured to generate a classification rule of the log according to the category data corresponding to the feature array and the log category.
  • the present application also provides a device for generating log classification rules.
  • the device for generating log classification rules includes a memory, a processor, and a device that is stored in the memory and can run on the processor.
  • a log classification rule generation program which implements the steps of the log classification rule generation method described above when the log classification rule generation program is executed by the processor.
  • this application also provides a computer-readable storage medium that stores a log classification rule generation program, which is implemented when the log classification rule generation program is executed by a processor The steps of the method for generating log classification rules as described above.
  • This application determines the log set corresponding to each log category in the log to be classified after obtaining the log to be classified, reads a log array from the log set to determine the target array, and determines the target array based on the target array and other log arrays in the log set. The first difference between the two determines the first information template corresponding to the log set. If only one type of first information template is detected, the feature array corresponding to the log set is determined according to the first information template, and the corresponding feature array and log category Classification rules for category data generation logs. In this embodiment, based on the concept of log arrays and templates, feature arrays are derived, and classification rules for logs are generated according to the feature arrays and log categories.
  • FIG. 1 is a schematic flowchart of a first embodiment of a method for generating a log classification rule of this application
  • FIG. 2 is a schematic flowchart of a second embodiment of a method for generating a log classification rule of this application
  • FIG. 3 is a schematic flowchart of a fourth embodiment of a method for generating log classification rules according to the present application
  • FIG. 4 is a functional schematic diagram module diagram of a preferred embodiment of an apparatus for generating log classification rules of the present application
  • Fig. 5 is a schematic structural diagram of a hardware operating environment involved in a solution of an embodiment of the present application.
  • FIG. 1 is a schematic flowchart of a first embodiment of a method for generating log classification rules in this application.
  • the embodiment of the application provides an embodiment of a method for generating a log classification rule. It should be noted that although the logical sequence is shown in the flowchart, in some cases, the sequence shown here can be executed in a different order. Steps out or described.
  • the method for generating log classification rules is applied to the device for generating log classification rules.
  • the generating devices can include mobile terminals such as tablet computers, notebook computers, handheld computers, and personal digital assistants (PDAs), as well as mobile terminals such as digital TVs and desktops. Fixed terminals such as computers.
  • PDAs personal digital assistants
  • Fixed terminals such as computers.
  • the execution subject is omitted to describe each embodiment.
  • the methods for generating log classification rules include:
  • Step S10 When the log to be classified is obtained, the log category corresponding to the log to be classified is determined, and the log set corresponding to each log category is determined.
  • the logs to be classified can be obtained from the web log server.
  • the network log server stores logs generated by network devices such as switches, routers, and firewalls during operation.
  • Methods for obtaining logs to be classified from the web log server include, but are not limited to, full synchronization methods, incremental synchronization methods, and conditional synchronization methods.
  • the full synchronization method is to obtain all the logs stored by the network log server as logs to be classified; the incremental synchronization method is to obtain the newly stored logs from the network log server as the logs to be classified.
  • the conditional synchronization method is to obtain logs stored in a specific time period from the network log server as logs to be classified, obtain logs generated by one or more devices as logs to be classified, or obtain logs generated by one or more device models corresponding to the device
  • the log serves as the log to be classified. It is understandable that one device corresponds to only one device model, but one device model can correspond to multiple devices.
  • the logs to be classified from the network-log server when obtaining the logs to be classified from the network-log server, it can be based on the file size, the number of files and the available network log server.
  • the bandwidth limits the rate at which logs to be classified can be obtained from the network log server, where the file size is the size of a log to be classified.
  • the file size and number of files can be used to determine how many logs to be classified can be obtained from the network log server each time under the current bandwidth, so that the network log server can be quickly retrieved from the network log server without affecting the normal operation of the network log server. Get the logs to be classified.
  • the log category corresponding to the log to be classified is determined, and the log set corresponding to each log category is determined.
  • step S10 includes:
  • Step a After the log to be classified is obtained, an information unit in each log line of the log to be classified is determined as an element to obtain an element array corresponding to each log line in the log to be classified.
  • Step b After stripping the timestamp element, the device name element and the log category identification element from the element array, the log array corresponding to each line of the log in the log to be classified is obtained.
  • the log line composed of the smallest information unit in the log to be classified is determined.
  • English words are used as the smallest information unit, and one information in each line of the log to be classified
  • the unit is determined to be an element, and all elements in each log line correspond to an element array corresponding to the log line.
  • the element array is to divide each log line according to a custom delimiter (usually a space character), and separate words as array elements, and store the elements in the array in order to form an element array. That is, the separator is used to divide each log line in the log to be classified according to the minimum information unit, and the element corresponding to each log line is obtained to obtain the element array corresponding to each line of the log.
  • the log array corresponding to each line of the log in the log to be classified is obtained. It is understandable that there are no timestamps, device names, and log category identifiers in the log array.
  • the separator can be set according to specific needs, for example, the separator can be set to a space character, or set to "
  • the time stamp is the time stamp of the generation time of the generated log.
  • the device name is the name of the device corresponding to the log to be classified.
  • the log category identifier is the log category identifier corresponding to the device that generates the log to be classified. The log category identifier is when the device is shipped from the factory. Once set, each device has its own log category identification.
  • Step c Determine the logs to be classified with the same device name and the log category identifier as logs belonging to the same log category.
  • Step d in the log arrays corresponding to the logs of the same log category, combine the log arrays with the same array length into a log set to obtain a log set corresponding to each log category.
  • the logs to be classified with the same device name and log category identifier are determined to belong to the same log category, and in the log arrays corresponding to the logs of the same log category, the arrays with the same length
  • the log arrays are combined into a log collection to obtain a log collection corresponding to each log category.
  • the number of elements in the log array with the same array length is the same. It should be noted that in order to reduce the number of log arrays in the log collection and improve the efficiency of obtaining feature arrays, it is necessary to perform deduplication operations on the log collection to delete redundant and identical log arrays in the log collection, so that every There is only one log array.
  • the log category identification is %LINK-3-UPDOWN and the device name is SZ-XL-0107-G05-ISR4451-IE-AR-01.
  • the corresponding log collection can be: [(Interface), (GigabitEthernet0/0/1,) , (Change), (state), (to), (down)], [(Interface), (GigabitEthernet0/0/1,), (change), (state), (to), (up)], [ (Interface), (GigabitEthernet1/0/1,), (change), (state), (to), (down)], [(Interface), (GigabitEthernet1/0/1,), (change), (state ), (to), (up)].
  • Step S20 Read a log array from the log set to determine a target array, and calculate a first difference between the target array and other log arrays in the log set.
  • Step S30 Determine a first information template corresponding to the log set according to the first difference, and detect whether there is only one type of the first information template.
  • a log array is read from the log set to determine the target array, and the first difference between the target array and other log arrays in the log set is calculated.
  • the target array can be the first array in the log collection or any array in the log collection.
  • it compares whether the elements at the same position in the log array and other log arrays in the log collection are the same. If the elements at the same position are the same, the corresponding position is calculated The result obtained is 0. If the elements at the same position are not the same, the calculated result of the corresponding position is 1.
  • the first information template corresponding to the log set is determined according to the first difference, and it is detected whether there is only one first information template.
  • the template is a special array.
  • the value of the array element is 0 or 1.
  • the information template is used to extract the changed information in similar log events. It is one of the key steps of semi-structured processing.
  • Information template and features The template is complementary, that is, the 0 and 1 elements of the information template are exchanged to become the corresponding feature template, and the information template can be obtained through the difference.
  • Semi-structured data is a form of structured data.
  • Step S40 If it is detected that there is only one type of the first information template, the feature array corresponding to the log set is determined according to the first information template, and the feature array and the category data corresponding to the log category are determined. Classification rules for generating logs.
  • the feature template corresponding to the log collection is determined according to the first information template
  • the feature array corresponding to the log collection is determined according to the feature template
  • the feature array and log category are determined
  • the category data includes, but is not limited to, the device name and log category identification.
  • the feature array and the category data corresponding to the log category are stored in association, that is, the classification rule of the log is generated, that is, in the classification rule, the category data is stored in association with the feature array, and a set of category data can correspond to one or more feature arrays.
  • the category to which the log belongs can be determined first based on the category data, and then the log corresponding to the category can be semi-structured according to the feature array corresponding to the category.
  • the feature template is a log array of the same log category, with the same number of elements and the same elements at specific locations. Replace the elements at these specific locations with 1, and replace the remaining locations with 0 to form the feature template.
  • the feature template is to convert the elements that do not contain information in the information template into 1 element, that is, the 0 element in the information template is correspondingly converted into 1 element, and the 1 element containing information is converted into 0 element.
  • the feature array is a special array.
  • the value of the array element is 0 or word. It is the result of extracting the log array from the feature template and reflects the word distribution characteristics of the log.
  • the feature template corresponding to the first information template [0, 1, 0, 0, 0, 1] is: [1, 0, 1, 1, 1, 0], and the corresponding feature array is [(Interface), 0, (changed), (state), (to), 0].
  • the log set corresponding to each log category in the log to be classified is determined, a log array is read from the log set to determine the target array, and the target array and other log arrays in the log set are determined.
  • the first difference between determines the first information template corresponding to the log set. If only one first information template is detected, the feature array corresponding to the log set is determined according to the first information template, and the feature array corresponds to the log category
  • the classification rules of the log generated from the category data. In this embodiment, based on the concept of log arrays and templates, feature arrays are derived, and classification rules for logs are generated according to the feature arrays and log categories.
  • step S40 includes:
  • Step e If it is detected that there is only one type of the first information template, determine the feature template corresponding to the first information template.
  • Step f Generate a feature array corresponding to the log set according to the feature template, add corresponding information tags to the information elements in the feature array, and add corresponding information tags to the feature array after adding the information tags and the log category Classification rules for category data generation logs.
  • an information element is an element containing information in a feature array.
  • an element with a value of 0 is recorded as an information element, and the added information label is determined by the meaning of the information element. For example, when the information element represents an IP address, an address tag can be added to the information element.
  • XML Extensible Markup Language, Extensible Markup Language
  • the log generated during the process of generating the log classification rule may be stored.
  • the classification rule can be output for the user to view, and the user can also adjust the classification rule according to specific needs.
  • Unstructured data is data that has irregular or incomplete data structure, does not have a predefined data model, and is inconvenient to use two-dimensional database tables to represent data, such as various types of text.
  • the generating device may be a financial institution device, and the financial institution device can classify the logs generated during the data processing of the financial institution device through the log classification rules generated in this embodiment.
  • the difference between the second embodiment of the method for generating log classification rules and the first embodiment of the method for generating log classification rules is that, referring to FIG. 2, the method for generating log classification rules further includes:
  • Step S50 if it is detected that there are at least two kinds of the first information templates, sequentially read other log arrays in the log set except the target array as the arrays to be calculated.
  • first information templates If it is detected that there are at least two types of first information templates, then sequentially read other data in the log set except the target array as the array to be calculated. It should be noted that in this embodiment, in the process of sequentially reading log arrays other than the target array in the log collection as the array to be calculated, it is not limited to reading from the front to the back in the log collection, or from the back in the log collection. To start reading before, just read all the log arrays except the target array in the log collection.
  • Step S60 Calculate the second difference between the array to be calculated and other log arrays in the log set except the array to be calculated.
  • Step S70 Determine a second information template corresponding to the array to be calculated according to the second difference, and calculate the number of occurrences of the second information template.
  • the second difference After reading the to-be-calculated array in the log collection, calculate the second difference between the to-be-calculated array and other log arrays in the log collection except the to-be-calculated array. It should be noted that the process of calculating the second difference It is similar to the process of calculating the first difference, and will not be repeated here. After the second difference is calculated, the second information template corresponding to the array to be calculated is determined according to the second difference, and the number of occurrences of the second information template is calculated. It should be noted that when the second difference value corresponds to at least two second information templates, the number of occurrences is the number of times corresponding to the information template with the most occurrences.
  • the number of occurrences of the A information template is 2, and the number of occurrences of the B information template is 3.
  • What needs to be used is the number of appearances of the B information template to be compared with the number of appearances of the first information template.
  • Step S80 If the number of appearances of the second information template is greater than the number of appearances of the first information template, it is detected whether the array to be calculated is the last array in the log set.
  • the number of appearances of the second information template After the number of appearances of the second information template is calculated, it is determined whether the number of appearances of the second information template is greater than the number of appearances of the first information template. If the number of appearances of the second information template is greater than the number of appearances of the first information template, the second information template is stored, and it is checked whether the array to be calculated is the last array read in the log set. Further, if the number of appearances of the second information template is less than or equal to the number of appearances of the first information template, it is detected whether the array to be calculated is the last array read in the log set.
  • Step S90 If the array to be calculated is the last array in the log set, determine the feature array corresponding to the log set according to the second information template, and correspond to the feature array and the log category The classification rules of the log generated from the category data.
  • the feature array corresponding to the log collection is determined according to the second information template, and the classification of the log is generated according to the determined feature array and the category data corresponding to the log category rule. It should be noted that the process of determining the feature array corresponding to the log set according to the second information template is similar to the process of determining the feature array corresponding to the log set according to the first information template, and will not be described in detail here.
  • the method for generating log classification rules further includes:
  • Step g if it is detected that the array to be calculated is not the last array in the log set, read a new log array from the log set to determine the target array, and execute the calculation of the target array The step of first difference with other log arrays in the log set.
  • the array to be calculated is not the last array in the log collection, read a new log array in the log collection to determine the target array, and calculate the first between the target array and other log arrays in the log collection Difference. It should be noted that there are at least two log arrays in the log collection, otherwise the information template cannot be obtained by calculating the difference.
  • the feature array corresponding to the log set is determined according to the second information template, and the classification rule of the log is generated according to the category data corresponding to the sign array and the log category; if it is detected that the array to be calculated is not the last array in the log set, the log Read a new log array from the collection to determine the target array, and re-calculate the first difference between the target array and the other log arrays in the log collection, so that the final log classification rule appears in the resulting log to be classified
  • the log classification rule with the highest probability improves the accuracy of the resulting log classification rule to classify logs.
  • step S40 further includes:
  • Step h If it is detected that there is only one type of the first information template, output a prompt message to prompt the user to confirm whether the first information template is correct.
  • Step i If a confirmation instruction to determine that the first information template is correct is received, determine the feature array corresponding to the log set according to the first information template, and according to the category corresponding to the feature array and the log category Classification rules for data generation logs.
  • a prompt message is generated, and the prompt message is output to prompt the user to confirm whether the first information template is correct, and it is detected whether a confirmation instruction to confirm that the first information template is correct is received.
  • the output manner of the prompt information is not limited. Specifically, the prompt information may be output by voice or text. If a confirmation instruction to determine that the first information template is correct is received, the feature array corresponding to the log set is determined according to the first information template, and a classification rule for the log is generated according to the category data corresponding to the feature array and the log category.
  • the first information template is modified according to the modification instruction to obtain the modified first information template, and the characteristics corresponding to the log set are determined according to the modified first information template Array. Further, if a confirmation instruction to determine that the first information template is wrong is received, the feature array corresponding to the log set is not determined according to the first information template. At this time, it can be determined that the generation of the log classification rule has failed.
  • a prompt message may also be output to prompt the user to confirm whether the second information template is correct.
  • the operation process after outputting the prompt message to prompt the user to confirm whether the second information template is correct is similar to the operation process after outputting the prompt message and the user confirms whether the first information template is correct, and will not be described in detail here.
  • a prompt message is output to prompt the user to confirm whether the first information template is correct. Only after receiving a confirmation instruction to determine that the first information template is correct, the log set corresponding to the first information template is determined Through the feature array of the feature array to generate log classification rules, the accuracy of semi-structural processing of log data by the generated classification rules is further improved, and the accuracy of log classification by the generated classification rules is improved.
  • the log classification rule generation method further includes:
  • Step S110 After acquiring a new log, extract category data in the new log, and determine a classification rule corresponding to the new log according to the category data.
  • the generated log is divided into rules, it is detected whether a new log is obtained in the network log server. If a new log is obtained from the web log server, the category data in the new log is extracted, and the classification rule corresponding to the new log is determined according to the category data. It should be noted that since there are category data in the classification rules, the classification rules corresponding to the new logs can be determined by comparing the category data extracted in the new log with the category data in the classification rules.
  • Step S120 Determine the feature array corresponding to the new log according to the classification rule corresponding to the new log, and process the new log according to the feature array corresponding to the new log to perform semi-structured processing on the new log.
  • the feature array corresponding to the new log is determined according to the classification rule corresponding to the new log.
  • a classification rule can correspond to one or more feature arrays. Therefore, in the process of determining the feature array corresponding to the new log according to the classification rule corresponding to the new log, the log array in the new log needs to be extracted, and the log array and the feature array need to be extracted. In contrast, if the log array of the new log is the same as other elements in a feature array except for the 0 element, it can be determined that the feature array is a feature array that matches the log array of the new log. It is understandable that the feature array that matches the log array of the new log is the feature array corresponding to the new log.
  • a new log may correspond to multiple log arrays, that is, a new log may correspond to multiple feature arrays. It should be noted that the process of determining the log array corresponding to the new log is the same as the process of determining the log array corresponding to each line of the log in the log to be classified in the first embodiment, and will not be described in detail here.
  • the new log is processed according to the feature array corresponding to the new log, that is, the feature array corresponding to the new log adds corresponding information tags to the information elements in the log array of the new log to perform half of the new log. Structured processing. It is understandable that after the semi-structured processing of the new log, since the elements containing specific information in the new log have been added with corresponding information tags, the subsequent data analysis on the new log can directly extract the information tags Corresponding log data to realize rapid analysis of new log data.
  • Steps S10 to S90 are used to train the classification rules corresponding to the new log.
  • the classification rule corresponding to the new log is determined, and then the feature array corresponding to the new log is determined according to the classification rule, the new log is processed according to the feature array corresponding to the new log, and the new log is processed Semi-structured processing, in order to semi-structure the log data in the process of log classification, improve the analysis efficiency of log data analysis.
  • the present application also provides a device for generating log classification rules.
  • the device for generating log classification rules includes:
  • the determining module 10 is configured to determine the log category corresponding to the log to be classified when the log to be classified is obtained, and determine the log set corresponding to each log category;
  • the reading module 20 is configured to read a log array from the log set and determine it as a target array
  • the calculation module 30 is configured to calculate the first difference between the target array and other log arrays in the log set;
  • the determining module 10 is further configured to determine a first information template corresponding to the log set according to the first difference;
  • the detection module 40 is configured to detect whether there is only one type of the first information template
  • the determining module 10 is further configured to determine the feature array corresponding to the log set according to the first information template if only one type of the first information template is detected;
  • the generating module 50 is configured to generate a log classification rule according to the feature array and the category data corresponding to the log category.
  • the reading module 20 is further configured to, if it is detected that there are at least two kinds of the first information templates, sequentially read other log arrays in the log set except the target array as the array to be calculated;
  • the calculation module 30 is also configured to calculate a second difference between the array to be calculated and other log arrays in the log set except the array to be calculated;
  • the determining module 10 is further configured to determine a second information template corresponding to the array to be calculated according to the second difference;
  • the calculation module 30 is also used to calculate the number of appearances of the second information template
  • the detection module 40 is further configured to detect whether the array to be calculated is the last array in the log set if the number of appearances of the second information template is greater than the number of appearances of the first information template;
  • the determining module 10 is further configured to determine the feature array corresponding to the log set according to the second information template if the array to be calculated is the last array in the log set.
  • the reading module 20 is further configured to read a new log array from the log set and determine it as a target array if it is detected that the array to be calculated is not the last array in the log set;
  • the generating device of the log classification rule includes:
  • the execution module is configured to execute the step of calculating the first difference between the target array and other log arrays in the log set.
  • the determining module 10 includes:
  • the first determining unit is configured to determine the feature template corresponding to the first information template if it is detected that there is only one type of the first information template;
  • a generating unit configured to generate a feature array corresponding to the log set according to the feature template
  • the adding unit is used to add corresponding information tags to the information elements in the feature array
  • the generating module 50 is further configured to generate a classification rule of the log according to the feature array after the information tag is added and the category data corresponding to the log category.
  • the determining module 10 further includes:
  • An output unit configured to output a prompt message to prompt the user to confirm whether the first information template is correct if it is detected that only one type of the first information template exists
  • a second determining unit configured to determine the feature array corresponding to the log set according to the first information template if a confirmation instruction for determining that the first information template is correct is received;
  • the generating module 50 is further configured to generate a log classification rule according to the category data corresponding to the feature array and the log category.
  • the determining module 10 includes:
  • the third determining unit is used for determining an information unit in each line of the log to be classified as an element after the log to be classified is obtained, so as to obtain an element array corresponding to each line of the log in the log to be classified ;
  • the processing unit is configured to extract the timestamp element, the device name element, and the log category identification element from the element array to obtain the log array corresponding to each line of the log in the log to be classified;
  • the third determining unit is further configured to determine the logs to be classified with the same device name and the log category identifier as logs belonging to the same log category;
  • the third determining unit is further configured to combine log arrays with the same array length in log arrays corresponding to logs of the same log category into a log set to obtain log sets corresponding to each log category.
  • the device for generating log classification rules further includes:
  • the extraction module is used to extract category data in the new log after the new log is obtained
  • the determining module 10 is further configured to determine the classification rule corresponding to the new log according to the category data; determine the feature array corresponding to the new log according to the classification rule corresponding to the new log;
  • the device for generating log classification rules further includes:
  • the processing module is configured to process the new log according to the feature array corresponding to the new log to perform semi-structured processing on the new log.
  • each embodiment of the device for generating a log classification rule is basically the same as each embodiment of the above-mentioned method for generating a log classification rule, and details are not described herein again.
  • FIG. 5 is a schematic structural diagram of the hardware operating environment involved in the solution of the embodiment of the present application.
  • FIG. 5 can be a structural diagram of the hardware operating environment of the device for generating log classification rules.
  • the device for generating the log classification rule in the embodiment of the present application may be a terminal device such as a PC and a portable computer.
  • the device for generating the log classification rule may include a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002.
  • the communication bus 1002 is used to implement connection and communication between these components.
  • the user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.
  • the network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).
  • the memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as a magnetic disk memory.
  • the memory 1005 may also be a storage device independent of the foregoing processor 1001.
  • the device for generating log classification rules may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and so on.
  • RF Radio Frequency
  • the structure of the device for generating log classification rules shown in FIG. 5 does not constitute a limitation on the device for generating log classification rules, and may include more or less components than shown in the figure, or a combination of some Components, or different component arrangements.
  • the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a log classification rule generation program.
  • the operating system is a program that manages and controls the hardware and software resources of the device that generates log classification rules, and supports the generation of log classification rules and the operation of other software or programs.
  • the user interface 1003 is mainly used for the network log server to communicate data with the network log server;
  • the network interface 1004 is mainly used for the background server to communicate data with the background server;
  • processor 1001 can be used to call the log classification rule generation program stored in the memory 1005 and execute the steps of the log classification rule generation method described above.
  • the specific implementation of the device for generating the log classification rule of this application is basically the same as the foregoing embodiments of the method for generating the log classification rule, and will not be repeated here.
  • an embodiment of the present application also proposes a computer-readable storage medium, the computer-readable storage medium stores a log classification rule generation program, and the log classification rule generation program is executed by a processor to achieve the above The steps of the method for generating log classification rules.
  • the method of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. ⁇
  • the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, The optical disc) includes several instructions to enable a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the method described in each embodiment of the present application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Debugging And Monitoring (AREA)

Abstract

一种日志分类规则的生成方法、装置、设备及可读存储介质，涉及金融科技领域，该方法包括步骤：当获取到待分类日志，确定待分类日志对应的日志类别，并确定各个日志类别对应的日志集合（S10）；在日志集合中读取一个日志数组确定为目标数组，并计算目标数组与日志集合中的其它日志数组之间的第一差值（S20）；根据第一差值确定日志集合对应的第一信息模板，并检测是否只存在一种第一信息模板（S30）；若检测到只存在一种第一信息模板，则根据第一信息模板确定日志集合对应的特征数组，并根据特征数组和日志类别对应的类别数据生成日志的分类规则（S40）。

Description

日志分类规则的生成方法、装置、设备及可读存储介质
本申请要求于2019年6月21日申请的、申请号为201910546292.1、名称为“日志分类规则的生成方法、装置、设备及可读存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及金融科技(Fintech)的数据处理技术领域，尤其涉及一种日志分类规则的生成方法、装置、设备及可读存储介质。
背景技术
随着金融科技，尤其是互联网科技金融的不断发展，越来越多的技术(如分布式、大数据、区块链Blockchain、人工智能等)应用在金融领域，但金融业也对技术提出了更高的要求，如在金融领域，当通过交换机、路由器和防火墙等实现线上金融交易时，需要对交易过程中产生的日志进行分类。
在线上交易过程中，在交换机、路由器和防火墙等网络设备中会产生文本记录，该文本记录就是日志。各网络设备厂商依据内部标准，将交换机、路由器和防火墙等网络设备运行期间遇到的事件记为日志，一般每一行日志描述某个时间点某个对象最新的状态。因为描述每类事件的句子格式不一致，且同类事件在不同设备型号上的句子格式也可能不一致，所以日志是结构不规则的非结构化数据。非结构化数据是数据结构不规则或不完整，没有预定义的数据模型，不方便用数据库二维逻辑表来表现的数据，比如各类文本。
目前常用的日志分类方法有如下两种，第一种是依据网络设备厂商和型号，人工分析网络设备的日志数据，然后定义每一类事件的正则表达式规则，由预先设置好的程序按照定义的正则表达式规则对日志进行分类。第二种是利用文本相似度算法计算每行日志样本之间的相似度，相似度大于设定值的为同类样本，利用同类样本生成分类所需的正则表达式规则，最后依据正则表达式规则对日志进行分类。第一种日志分类方法需要人工分析日志并定义分类所需的正则表达式，工作量大，日志分类成本高，正则表达式规则生成效率低下；第二种 日志分类方法的正则表达式来自日志样本，而分类依赖日志样本之间相似度的设定值，该方法只能通过设定值间接干预分类所需的正则表达式规则，生成正则表达式规则的过程繁琐。
由此可知，目前生成日志分类所需的正则表达式规则过程繁琐，效率低下。
发明概述
技术问题
问题的解决方案
技术解决方案
本申请的主要目的在于提供一种日志分类规则的生成方法、装置、设备及可读存储介质，旨在解决现有的生成日志分类所需的正则表达式规则过程繁琐，效率低下的技术问题。
为实现上述目的，本申请提供一种日志分类规则的生成方法，所述日志分类规则的生成方法包括步骤：
当获取到待分类日志，确定所述待分类日志对应的日志类别，并确定各个日志类别对应的日志集合；
在所述日志集合中读取一个日志数组确定为目标数组，并计算所述目标数组与所述日志集合中的其它日志数组之间的第一差值；
根据所述第一差值确定所述日志集合对应的第一信息模板，并检测是否只存在一种所述第一信息模板；
若检测到只存在一种所述第一信息模板，则根据所述第一信息模板确定所述日志集合对应的特征数组，并根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
此外，为实现上述目的，本申请还提供一种日志分类规则的生成装置，所述日志分类规则的生成装置包括：
确定模块，用于当获取到待分类日志，确定所述待分类日志对应的日志类别，并确定各个日志类别对应的日志集合；
读取模块，用于在所述日志集合中读取一个日志数组确定为目标数组；
计算模块，用于计算所述目标数组与所述日志集合中的其它日志数组之间的第 一差值；
所述确定模块还用于根据所述第一差值确定所述日志集合对应的第一信息模板；
检测模块，用于检测是否只存在一种所述第一信息模板；
所述确定模块还用于若检测到只存在一种所述第一信息模板，则根据所述第一信息模板确定所述日志集合对应的特征数组；
生成模块，用于根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
此外，为实现上述目的，本申请还提供一种日志分类规则的生成设备，所述日志分类规则的生成设备包括存储器、处理器和存储在所述存储器上并可在所述处理器上运行的日志分类规则的生成程序，所述日志分类规则的生成程序被所述处理器执行时实现如上所述的日志分类规则的生成方法的步骤。
此外，为实现上述目的，本申请还提供一种计算机可读存储介质，所述计算机可读存储介质上存储有日志分类规则的生成程序，所述日志分类规则的生成程序被处理器执行时实现如上所述的日志分类规则的生成方法的步骤。
本申请通过在获取到待分类日志后，确定待分类日志中各个日志类别对应的日志集合，在日志集合中读取一个日志数组确定为目标数组，根据目标数组与日志集合中的其它日志数组之间的第一差值确定日志集合对应的第一信息模板，若检测到只存在一种第一信息模板，则根据第一信息模板确定日志集合对应的特征数组，根据特征数组和日志类别对应的类别数据生成日志的分类规则。本实施例基于日志数组和模板的概念，得出特征数组，根据特征数组和日志类别生成日志的分类规则，通过模板和数组的直观性和可调节性便于用户快速调整分类规则，且不再需要生成正则表达式规则对日志进行分类，而是通过数组和类别数据生成日志的分类规则，在生成分类规则过程中，不需要人工分析日志，以及定义分类所需的正则表达式，简化了分类规则的生成流程，以及提高了分类规则的生成效率。
发明的有益效果
对附图的简要说明
附图说明
图1是本申请日志分类规则的生成方法第一实施例的流程示意图；
图2是本申请日志分类规则的生成方法第二实施例的流程示意图；
图3是本申请日志分类规则的生成方法第四实施例的流程示意图；
图4是本申请日志分类规则的生成装置较佳实施例的功能示意图模块图；
图5是本申请实施例方案涉及的硬件运行环境的结构示意图。
本申请目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。
发明实施例
本发明的实施方式
应当理解，此处所描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
本申请提供一种日志分类规则的生成方法，参照图1，图1为本申请日志分类规则的生成方法第一实施例的流程示意图。
本申请实施例提供了日志分类规则的生成方法的实施例，需要说明的是，虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于此处的顺序执行所示出或描述的步骤。
日志分类规则的生成方法应用于日志分类规则的生成设备中，生成设备可以包括诸如平板电脑、笔记本电脑、掌上电脑、个人数字助理(Personal Digital Assistant，PDA)等移动终端，以及诸如数字TV、台式计算机等固定终端。在日志分类规则的生成方法的各个实施例中，为了便于描述，省略执行主体进行阐述各个实施例。
日志分类规则的生成方法包括：
步骤S10，当获取到待分类日志，确定所述待分类日志对应的日志类别，并确定各个日志类别对应的日志集合。
在本申请实施例中，可从网络日志服务器获取待分类日志。在网络日志服务器中，存储有交换机、路由器和防火墙等网络设备运行时产生的日志。从网络日志服务器中获取待分类日志的方法包括但不限于全量同步方法、增量同步方法和条件同步方法。全量同步方法为获取网络日志服务器存储的所有日志作为待 分类日志；增量同步方法为从网络日志服务器中获取新存储的日志作为待分类日志，此时，待分类日志为未获取过的日志；条件同步方法为可从网络日志服务器中获取特定时间段存储的日志作为待分类日志，获取其中一个或者多个设备产生的日志作为待分类日志，或者获取其中一个或者多个设备型号对应设备产生的日志作为待分类日志。可以理解的是，一个设备只对应一个设备型号，但一个设备型号可对应多个设备。
进一步地，在获取待分类过程中，为了避免影响网络日志服务器的正常运行，在从网络日志服务器中获取待分类日志时，可根据待分类日志的文件大小、文件数量和网络日志服务器所能用的带宽限制从网络日志服务器中获取待分类日志的速率，其中，文件大小为一个待分类日志的大小。具体地，可根据文件大小和文件数量确定在当前带宽下，每次可从网络日志服务器中获取多少待分类日志，才可以在不影响网络日志服务器正常运行的情况下，快速从网络日志服务器中获取待分类日志。
当从网络日志服务器中获取到待分类日志后，确定待分类日志对应的日志类别，并确定各个日志类别对应的日志集合。
进一步地，步骤S10包括：
步骤a，当获取到待分类日志后，将所述待分类日志的每一行日志中的一个信息单元确定为一个元素，以得到所述待分类日志中每一行日志对应的元素数组。
步骤b，在所述元素数组中剥离出时间戳元素、设备名称元素和日志类别标识元素后，得到所述待分类日志中每一行日志对应的日志数组。
具体地，当获取到待分类日志后，确定待分类日志中最小信息单元构成的日志行，在本实施例中，以英文单词作为最小信息单元，将待分类日志的每一行日志中的一个信息单元确定为一个元素，每一行日志中的全部元素对应组成了该行日志对应的元素数组。元素数组是将每行日志按照自定义的分隔符(一般为空格符)，切分出单词作为数组元素，按顺序将元素存入数组构成元素数组。即采用分隔符，按照最小信息单元切分待分类日志中的每一行日志，得到每一行日志对应的元素，以得到每一行日志对应的元素数组。在元素数组中剥离出 时间戳元素、设备名称元素和日志类别标识元素后，得到待分类日志中每一行日志对应的日志数组。可以理解的是，日志数组中是不存在时间戳、设备名称和日志类别标识的。分隔符可根据具体需要而设置，如可将分隔符设置为空格符，或者设置为“|”等。时间戳是生成日志的生成时间的时间戳，设备名称是产生待分类日志对应的设备的名称，日志类别标识是产生待分类日志对应的设备对应的日志类别标识，日志类别标识是设备在出厂时就设定好的，每个设备都存在自己的日志类别标识。
如若某一行原始日志为：“2019 Mar 22 15：50：13.196 SZ-XL-0107-G05-ISR4451-IE-AR-01％LINK-3-UPDOWN：Interface GigabitEthernet0/0/1，change state to up”，对应的元素数组为：[(2019)，(Mar)，(22)，(15：50：13.196)，(SZ-XL-0107-G05-ISR4451-IE-AR-01)，(％LINK-3-UPDOWN：)，(Interface)，(GigabitEthernet0/0/1，)，(change)，(state)，(to)，(up)]，其中，[]表示数组，()表示元素，其中(％LINK-3-UPDOWN)是日志类别标识元素，(2019)，(Mar)，(22)，(15：50：13.196)都是时间戳元素，(SZ-XL-0107-G05-ISR4451-IE-AR-01)是设备名称元素。此时，对应日志数组为[(Interface)，(GigabitEthernet0/0/1，)，(change)，(state)，(to)，(up)]。
步骤c，将所述设备名称和所述日志类别标识相同的待分类日志确定为属于同一日志类别的日志。
步骤d，在同一日志类别的日志对应的日志数组中，将数组长度相同的日志数组组合成日志集合，以得到各个日志类别对应的日志集合。
当得到待分类日志对应的所有日志数组后，将设备名称和日志类别标识相同的待分类日志确定属于同一日志类别的日志，并在同一日志类别的日志对应的日志数组中，将数组长度相同的日志数组组合成日志集合，以得到各个日志类别对应的日志集合。其中，数组长度相同的日志数组中的元素个数是相同。需要说明的是，为了减少日志集合中日志数组的数量，提高得到特征数组的效率，需要对日志集合进行去重操作，删除日志集合中多余的，完全相同的日志数组 ，使日志集合中，每种日志数组都只存在一个。
如日志类别标识为％LINK-3-UPDOWN，设备名称为SZ-XL-0107-G05-ISR4451-IE-AR-01对应的日志集合可为：[(Interface)，(GigabitEthernet0/0/1，)，(change)，(state)，(to)，(down)]，[(Interface)，(GigabitEthernet0/0/1，)，(change)，(state)，(to)，(up)]，[(Interface)，(GigabitEthernet1/0/1，)，(change)，(state)，(to)，(down)]，[(Interface)，(GigabitEthernet1/0/1，)，(change)，(state)，(to)，(up)]。
步骤S20，在所述日志集合中读取一个日志数组确定为目标数组，并计算所述目标数组与所述日志集合中的其它日志数组之间的第一差值。
步骤S30，根据所述第一差值确定所述日志集合对应的第一信息模板，并检测是否只存在一种所述第一信息模板。
当得到待分类日志中各个日志类别对应的日志集合后，在日志集合中读取一个日志数组确定为目标数组，并计算该目标数组与日志集合中其它日志数组之间的第一差值。需要说明的是，目标数组可为日志集合中的第一个数组，也可为日志集合中的任意一个数组。在计算目标数组与日志集合中其它日志数组之间的第一差值过程中，是比较日志数组与日志集合中其它日志数组相同位置的元素是否相同，如果相同位置的元素相同，则对应位置计算所得的结果为0，如果相同位置的元素不相同，则对应位置计算所得的结果为1。
当计算得到目标数组与日志集合中的其它日志数组之间的第一差值后，根据第一差值确定日志集合对应的第一信息模板，并检测是否只存在一种第一信息模板。在本实施例中，模版是一种特殊数组，数组元素取值为0或1，信息模板用于提取同类日志事件里变化的信息，是半结构化处理的关键步骤之一，信息模板与特征模板互补，即将信息模板的0和1元素互换就成了对应的特征模板，通过求差可获得信息模板。半结构化数据是结构化数据的一种形式，它并不符合关系型数据库或其他数据表的形式关联起来的数据模型结构，但包含相关标记，用来分隔语义元素以及对记录和字段进行分层，因此，它也被称为自描述的结构。半结构化数据属于同一类实体可以有不同的属性，即使他们被组合在一起，这些属性的顺序并不重要。
若[(Interface)，(GigabitEthernet0/0/1，)，(change)，(state)，(to)，(down)]为目 标数组，则计算目标数组与[(Interface)，(GigabitEthernet1/0/1，)，(change)，(state)，(to)，(up)]之间的差值，所得的第一差值分别为0，1，0，0，0和1，此时，所得的第一信息模板为[0，1，0，0，0，1]。
步骤S40，若检测到只存在一种所述第一信息模板，则根据所述第一信息模板确定所述日志集合对应的特征数组，并根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
若检测到该日志集合只存在一种第一信息模板，则根据第一信息模板确定日志集合对应的特征模板，根据该特征模板确定该日志集合对应的特征数组，并根据该特征数组和日志类别对应的类别数据生成日志的分类规则。其中，类别数据包括但不限于设备名称和日志类别标识。将特征数组和日志类别对应的类别数据关联存储，即生成了日志的分类规则，即在分类规则中，类别数据是和特征数组关联存储的，一组类别数据可对应一个或者多个特征数组。在所生成的分类规则中，首先根据类别数据可确定日志所属的类别，然后根据该类别对应的特征数组可为该类别对应的日志进行半结构化处理。
需要说明的是，特征模板是同日志类别的日志数组，元素数量相同且特定位置元素一样，将这些特定位置的元素换成1，其余位置换成0就构成了特征模板。特征模板是将信息模板中不含信息的元素转换成1元素，即将信息模板中0元素对应转换成1元素，将含有信息的1元素转换成0元素。特征数组是一种特殊数组，数组元素取值为0或单词，是特征模版提取日志数组后产生的结果，反映了日志的单词分布特征。如第一信息模板[0，1，0，0，0，1]对应的特征模板为：[1，0，1，1，1，0]，对应的特征数组为[(Interface)，0，(changed)，(state)，(to)，0]。
本实施例通过在获取到待分类日志后，确定待分类日志中各个日志类别对应的日志集合，在日志集合中读取一个日志数组确定为目标数组，根据目标数组与日志集合中的其它日志数组之间的第一差值确定日志集合对应的第一信息模板，若检测到只存在一种第一信息模板，则根据第一信息模板确定日志集合对应的特征数组，根据特征数组和日志类别对应的类别数据生成日志的分类规则。本实施例基于日志数组和模板的概念，得出特征数组，根据特征数组和日志类 别生成日志的分类规则，通过模板和数组的直观性和可调节性便于用户快速调整分类规则，且不再需要生成正则表达式规则对日志进行分类，而是通过数组和类别数据生成日志的分类规则，在生成分类规则过程中，不需要人工分析日志，以及定义分类所需的正则表达式，简化了分类规则的生成流程，以及提高了分类规则的生成效率。
需要说明的是，现有的通过正则表达式规则对日志分类过程中，不能对日志的数据进行处理，即日志中的数据还是非结构化数据，降低了用户通过日志进行数据分析和数据统计的效率。进一步地，步骤S40包括：
步骤e，若检测到只存在一种所述第一信息模板，则确定所述第一信息模板对应的特征模板。
步骤f，根据所述特征模板生成所述日志集合对应的特征数组，为所述特征数组中的信息元素添加对应的信息标签，根据添加信息标签后的所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
进一步地，若检测到只存在一种第一信息模板，则确定第一信息模板对应的特征模板，根据特征模板生成日志集合对应的特征数组，为特征数组中的信息元素添加对应的信息标签，并根据添加信息标签后的特征数组和日志类别对应的类别数据生成日志的分类规则。需要说明的是，信息元素为特征数组中含有信息的元素，在特征数组中，将数值为0的元素记为信息元素，所添加的信息标签由该信息元素表示的意思来确定。如当该信息元素表示的是IP地址时，可为该信息元素添加地址标签。在本实施例中，采用XML(Extensible Markup Language，可扩展标记语言)格式存储特征数组。
进一步地，为了便于用户查看日志分类规则的生成过程，在生成日志的分类规则过程中，可存储该生成日志分类规则过程中产生的日志。进一步地，为了便于用户查看所生成的日志分类规则，可输出该分类规则供用户查看，用户也可根据具体需要调整该分了规则。
通过为特征数组中携带信息的信息元素添加对应的标签，实现了将非结构化数据转换成半结构化数据，以通过半结构化数据提高了数据统计、数据分析和数据异常跟进的效率。非结构化数据是数据结构不规则或不完整，没有预定义的 数据模型，不方便用数据库二维逻辑表来表现的数据，比如各类文本。
需要说明的是，该生成设备可为金融机构设备，金融机构设备可通过本实施例生成的日志分类规则，对金融机构设备数据处理过程中生成的日志进行分类处理。
进一步地，提出本申请日志分类规则的生成方法第二实施例。
所述日志分类规则的生成方法第二实施例与所述日志分类规则的生成方法第一施例的区别在于，参照图2，日志分类规则的生成方法还包括：
步骤S50，若检测到存在至少两种所述第一信息模板，则依次读取所述日志集合中除所述目标数组外的其它日志数组作为待计算数组。
若检测到存在至少两种第一信息模板，则依次读取日志集合中除目标数组外的其它数据作为待计算数组。需要说明的是，本实施例中依次读取日志集合中除目标数组外的其它日志数组作为待计算数组过程中，不限于从在日志集合中从前到后开始读，还是从日志集合中从后到前开始读，只要将日志集合中除目标数组外的其它日志数组都读取到即可。
步骤S60，计算所述待计算数组与所述日志集合中除所述待计算数组外的其它日志数组之间的第二差值。
步骤S70，根据所述第二差值确定所述待计算数组对应的第二信息模板，并计算所述第二信息模板的出现次数。
当在日志集合中读取到待计算数组后，计算待计算数组与日志集合中除待计算数组外的其它日志数组之间的第二差值，需要说明的是，计算第二差值的过程与计算第一差值的过程类似，在此不再重复赘述。当计算出第二差值后，根据第二差值确定待计算数组对应的第二信息模板，以及计算第二信息模板的出现次数。需要说明的是，当第二差值对应至少两个第二信息模板时，出现次数是出现次数最多的信息模板对应的次数。如第二差值对应的第二信息模板有两个，分别为A信息模板和B信息模板，而A信息模板的出现次数为2，B信息模板的出现次数为3，则本实施例中所需用到的，与第一信息模板的出现次数进行比较的是B信息模板的出现次数。
步骤S80，若所述第二信息模板的出现次数大于所述第一信息模板的出现次数 ，则检测所述待计算数组是否为所述日志集合中的最后一个数组。
当计算得到第二信息模板的出现次数后，判断第二信息模板的出现次数是否大于第一信息模板的出现次数。若第二信息模板的出现次数大于第一信息模板的出现次数，则存储第二信息模板，并检测待计算数组是否为日志集合中被读取的最后一个数组。进一步地，若第二信息模板的出现次数小于或者等于第一信息模板的出现次数，则检测待计算数组是否为日志集合中被读取的最后一个数组。
步骤S90，若所述待计算数组为所述日志集合中的最后一个数组，则根据所述第二信息模板确定所述日志集合对应的特征数组，并根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
若检测到待计算数组为日志集合中被读取的最后一个数组，则根据第二信息模板确定日志集合对应的特征数组，并根据所确定的特征数组和日志类别对应的类别数据生成日志的分类规则。需要说明的是，在根据第二信息模板确定日志集合对应的特征数组的过程和根据第一信息模板确定日志集合对应的特征数组的过程类似，在此不再详细赘述。
进一步地，日志分类规则的生成方法还包括：
步骤g，若检测到所述待计算数组不是所述日志集合中的最后一个数组，则在所述日志集合中读取一个新的日志数组确定为目标数组，并执行所述计算所述目标数组与所述日志集合中的其它日志数组之间的第一差值的步骤。
若检测到待计算数组不是日志集合中的最后一个数组，则在日志集合中读取一个新的日志数组确定为目标数组，并计算该目标数组与日志集合中其它的日志数组之间的第一差值。需要说明的是，日志集合中的日志数组至少为两个，否者无法通过计算差值得到信息模板。
本实施例通过在检测到存在至少两种第一信息模板后，依次读取日志集合中除目标数组外的其它日志数组作为待计算数组，根据待计算数组与日志集合中除待计算数组外的其它日志数组之间的第二差值确定待计算数组对应的第二信息模板，若第二信息模板的出现次数大于第一信息模板的出现次数，且待计算数组为日志集合中的最后一个数组，则根据第二信息模板确定日志集合对应的特 征数组，并根据征数组和日志类别对应的类别数据生成日志的分类规则；若检测到待计算数组不是日志集合中的最后一个数组，则在日志集合中读取一个新的日志数组确定为目标数组，并重新执行计算目标数组与日志集合中的其它日志数组之间的第一差值，使最终所得的日志分类规则为所得待分类日志中出现概率最高的日志分类规则，提高了所得的日志分类规则分类日志的准确率。
进一步地，提出本申请日志分类规则的生成方法第三实施例。
所述日志分类规则的生成方法第三实施例与所述日志分类规则的生成方法第一和/或第二实施例的区别在于，步骤S40还包括：
步骤h，若检测到只存在一种所述第一信息模板，则输出提示信息提示用户确认所述第一信息模板是否正确。
步骤i，若接收到确定所述第一信息模板正确的确认指令，则根据所述第一信息模板确定所述日志集合对应的特征数组，并根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
若检测到只存在一种第一信息模板，则生成提示信息，并输出该提示信息提示用户确认第一信息模板是否正确，并检测是否接收到确定第一信息模板正确的确认指令。在本实施例中，不限制提示信息的输出方式，具体地，可通过语音或者文字等方式输出提示信息。若接收到确定第一信息模板正确的确认指令，则根据该第一信息模板确定日志集合对应的特征数组，并根据该特征数组和日志类别对应的类别数据生成日志的分类规则。进一步地，若接收到修改第一信息模板的修改指令，则根据该修改指令修改第一信息模板，得到修改后的第一信息模板，并根据修改后的第一信息模板确定日志集合对应的特征数组。进一步地，若接收到确定第一信息模板错误的确认指令，则不根据该第一信息模板确定日志集合对应的特征数组，此时，可确定日志的分类规则生成失败。
可以理解的是，在根据第二信息模板确定日志集合对应的特征数组之前，也可以输出提示信息提示用户确认第二信息模板是否正确。在输出提示信息提示用户确认第二信息模板是否正确之后的操作过程与输出提示信息用户确认第一信息模板是否正确之后的操作过程类似，在此不再详细赘述。
本实施例通过在得到第一信息模板后，输出提示信息提示用户确认第一信息模 板是否正确，只有在接收到确定第一信息模板正确的确认指令后，才根据第一信息模板确定日志集合对应的特征数组，通过该特征数组生成日志的分类规则，进一步提高了所生成的分类规则对日志数据进行半结构处理的准确率，以及提高了所生成的分类规则对日志分类的准确率。
进一步地，提出本申请日志分类规则的生成方法第四实施例。
所述日志分类规则的生成方法第四实施例与所述日志分类规则的生成方法第一、第二和/或第三实施例的区别在于，参照图3，日志分类规则的生成方法还包括：
步骤S110，当获取到新日志后，提取所述新日志中的类别数据，根据所述类别数据确定所述新日志对应的分类规则。
在生成日志分了规则后，检测是否在网络日志服务器中获取到新日志。若在网络日志服务器中获取到新日志，则提取新日志中的类别数据，并根据该类别数据确定新日志对应的分类规则。需要说明的是，由于在分类规则中，是存在类别数据的，因此，将新日志中提取的类别数据与分类规则中的类别数据进行对比，即可确定新日志对应的分类规则。
步骤S120，根据所述新日志对应的分类规则确定所述新日志对应的特征数组，根据所述新日志对应的特征数组处理所述新日志，以对所述新日志进行半结构化处理。
当确定新日志对应的分类规则后，根据新日志对应的分类规则确定新日志对应的特征数组。其中，一个分类规则可对应一个或者多个特征数组，因此，在根据新日志对应的分类规则确定新日志对应的特征数组过程中，需要提取新日志中的日志数组，将日志数组与特征数组进行对比，若新日志的日志数组与某个特征数组中除0元素外的其它元素都相同，则可确定该特征数组为与新日志的日志数组匹配的特征数组。可以理解的是，与新日志的日志数组匹配的特征数组为与新日志对应的特征数组。在一个新日志中，会存在多行日志，因此一个新日志会可能会对应多个日志数组，即一个新日志可能对应多个特征数组。需要说明的是，确定新日志对应的日志数组过程和第一实施例中确定待分类日志中每一行日志对应的日志数组的过程相同，在此不再详细赘述。
当确定新日志对应的特征数组后，根据新日志对应的特征数组处理新日志，即新日志对应的特征数组为新日志的日志数组中的信息元素添加对应的信息标签，以对新日志进行半结构化处理。可以理解的是，当对新日志进行半结构化处理后，由于新日志中含有特定信息的元素已被添加对应的信息标签，因此后续在对该新日志进行数据分析，可直接提取该信息标签对应的日志数据，以实现对新日志数据的快速分析。
进一步地，若新日志无法匹配分类规则，或者匹配分类规则后无法匹配出对应的特征数组，则存储该新日志，并输出通知消息通知对应的用户新日志无法分类，或者根据该新日志重新执行步骤S10至步骤S90，以训练出该新日志对应的分类规则。
本实施例通过在获取到新日志后，确定该新日志对应的分类规则，然后根据该分类规则确定新日志对应的特征数组，根据新日志对应的特征数组处理该新日志，对该新日志进行半结构化处理，以在对日志分类过程中对日志数据进行半结构处理，提高了对日志数据分析的分析效率。
此外，参照图4，本申请还提供一种日志分类规则的生成装置，所述日志分类规则的生成装置包括：
确定模块10，用于当获取到待分类日志，确定所述待分类日志对应的日志类别，并确定各个日志类别对应的日志集合；
读取模块20，用于在所述日志集合中读取一个日志数组确定为目标数组；
计算模块30，用于计算所述目标数组与所述日志集合中的其它日志数组之间的第一差值；
所述确定模块10还用于根据所述第一差值确定所述日志集合对应的第一信息模板；
检测模块40，用于检测是否只存在一种所述第一信息模板；
所述确定模块10还用于若检测到只存在一种所述第一信息模板，则根据所述第一信息模板确定所述日志集合对应的特征数组；
生成模块50，用于根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
进一步地，所述读取模块20还用于若检测到存在至少两种所述第一信息模板，则依次读取所述日志集合中除所述目标数组外的其它日志数组作为待计算数组；
所述计算模块30还用于计算所述待计算数组与所述日志集合中除所述待计算数组外的其它日志数组之间的第二差值；
所述确定模块10还用于根据所述第二差值确定所述待计算数组对应的第二信息模板；
所述计算模块30还用于计算所述第二信息模板的出现次数；
所述检测模块40还用于若所述第二信息模板的出现次数大于所述第一信息模板的出现次数，则检测所述待计算数组是否为所述日志集合中的最后一个数组；
所述确定模块10还用于若所述待计算数组为所述日志集合中的最后一个数组，则根据所述第二信息模板确定所述日志集合对应的特征数组。
进一步地，所述读取模块20还用于若检测到所述待计算数组不是所述日志集合中的最后一个数组，则在所述日志集合中读取一个新的日志数组确定为目标数组；
所述日志分类规则的生成装置包括：
执行模块，用于执行所述计算所述目标数组与所述日志集合中的其它日志数组之间的第一差值的步骤。
进一步地，所述确定模块10包括：
第一确定单元，用于若检测到只存在一种所述第一信息模板，则确定所述第一信息模板对应的特征模板；
生成单元，用于根据所述特征模板生成所述日志集合对应的特征数组；
添加单元，用于为所述特征数组中的信息元素添加对应的信息标签；
所述生成模块50还用于根据添加信息标签后的所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
进一步地，所述确定模块10还包括：
输出单元，用于若检测到只存在一种所述第一信息模板，则输出提示信息提示用户确认所述第一信息模板是否正确；
第二确定单元，用于若接收到确定所述第一信息模板正确的确认指令，则根据所述第一信息模板确定所述日志集合对应的特征数组；
所述生成模块50还用于根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
进一步地，所述确定模块10包括：
第三确定单元，用于当获取到待分类日志后，将所述待分类日志的每一行日志中的一个信息单元确定为一个元素，以得到所述待分类日志中每一行日志对应的元素数组；
处理单元，用于在所述元素数组中剥离出时间戳元素、设备名称元素和日志类别标识元素后，得到所述待分类日志中每一行日志对应的日志数组；
所述第三确定单元还用于将所述设备名称和所述日志类别标识相同的待分类日志确定为属于同一日志类别的日志；
所述第三确定单元还用于在同一日志类别的日志对应的日志数组中，将数组长度相同的日志数组组合成日志集合，以得到各个日志类别对应的日志集合。
进一步地，所述日志分类规则的生成装置还包括：
提取模块，用于当获取到新日志后，提取所述新日志中的类别数据；
所述确定模块10还用于根据所述类别数据确定所述新日志对应的分类规则；根据所述新日志对应的分类规则确定所述新日志对应的特征数组；
所述日志分类规则的生成装置还包括：
处理模块，用于根据所述新日志对应的特征数组处理所述新日志，以对所述新日志进行半结构化处理。
需要说明的是，日志分类规则的生成装置的各个实施例与上述日志分类规则的生成方法的各实施例基本相同，在此不再详细赘述。
此外，本申请还提供一种日志分类规则的生成设备。如图5所示，图5是本申请实施例方案涉及的硬件运行环境的结构示意图。
需要说明的是，图5即可为日志分类规则的生成设备的硬件运行环境的结构示意图。本申请实施例日志分类规则的生成设备可以是PC，便携计算机等终端设备。
如图5所示，该日志分类规则的生成设备可以包括：处理器1001，例如CPU，存储器1005，用户接口1003，网络接口1004，通信总线1002。其中，通信总线1002用于实现这些组件之间的连接通信。用户接口1003可以包括显示屏(Display)、输入单元比如键盘(Keyboard)，可选用户接口1003还可以包括标准的有线接口、无线接口。网络接口1004可选的可以包括标准的有线接口、无线接口(如WI-FI接口)。存储器1005可以是高速RAM存储器，也可以是稳定的存储器(non-volatile memory)，例如磁盘存储器。存储器1005可选的还可以是独立于前述处理器1001的存储装置。
可选地，日志分类规则的生成设备还可以包括摄像头、RF(Radio Frequency，射频)电路，传感器、音频电路、WiFi模块等等。
本领域技术人员可以理解，图5中示出的日志分类规则的生成设备结构并不构成对日志分类规则的生成设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。
如图5所示，作为一种计算机存储介质的存储器1005中可以包括操作系统、网络通信模块、用户接口模块以及日志分类规则的生成程序。其中，操作系统是管理和控制日志分类规则的生成设备硬件和软件资源的程序，支持日志分类规则的生成程序以及其它软件或程序的运行。
在图5所示的日志分类规则的生成设备中，用户接口1003主要用于网络日志服务器，与网络日志服务器进行数据通信；网络接口1004主要用于后台服务器，与后台服务器进行数据通信；处理器1001可以用于调用存储器1005中存储的日志分类规则的生成程序，并执行如上所述的日志分类规则的生成方法的步骤。
本申请日志分类规则的生成设备具体实施方式与上述日志分类规则的生成方法各实施例基本相同，在此不再赘述。
此外，本申请实施例还提出一种计算机可读存储介质，所述计算机可读存储介质上存储有日志分类规则的生成程序，所述日志分类规则的生成程序被处理器执行时实现如上所述的日志分类规则的生成方法的步骤。
本申请计算机可读存储介质具体实施方式与上述日志分类规则的生成方法各实施例基本相同，在此不再赘述。
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个......”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
上述本申请实施例序号仅仅为了描述，不代表实施例的优劣。
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，空调器，或者网络设备等)执行本申请各个实施例所述的方法。
以上仅为本申请的优选实施例，并非因此限制本申请的专利范围，凡是利用本申请说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本申请的专利保护范围内。

Claims (12)

  1. 一种日志分类规则的生成方法，其中，所述日志分类规则的生成方法包括以下步骤：
    当获取到待分类日志，确定所述待分类日志对应的日志类别，并确定各个日志类别对应的日志集合；
    在所述日志集合中读取一个日志数组确定为目标数组，并计算所述目标数组与所述日志集合中的其它日志数组之间的第一差值；
    根据所述第一差值确定所述日志集合对应的第一信息模板，并检测是否只存在一种所述第一信息模板；
    若检测到只存在一种所述第一信息模板，则根据所述第一信息模板确定所述日志集合对应的特征数组，并根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
  2. 如权利要求1所述的日志分类规则的生成方法，其中，所述根据所述第一差值确定所述日志集合对应的第一信息模板，并检测是否只存在一种所述第一信息模板的步骤之后，还包括：
    若检测到存在至少两种所述第一信息模板，则依次读取所述日志集合中除所述目标数组外的其它日志数组作为待计算数组；
    计算所述待计算数组与所述日志集合中除所述待计算数组外的其它日志数组之间的第二差值；
    根据所述第二差值确定所述待计算数组对应的第二信息模板，并计算所述第二信息模板的出现次数；
    若所述第二信息模板的出现次数大于所述第一信息模板的出现次数，则检测所述待计算数组是否为所述日志集合中的最后一个数组；
    若所述待计算数组为所述日志集合中的最后一个数组，则根据所述第二信息模板确定所述日志集合对应的特征数组，并根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
  3. 如权利要求2所述的日志分类规则的生成方法，其中，所述检测所 述待计算数组是否为所述日志集合中的最后一个数组的步骤之后，还包括：
    若检测到所述待计算数组不是所述日志集合中的最后一个数组，则在所述日志集合中读取一个新的日志数组确定为目标数组，并执行所述计算所述目标数组与所述日志集合中的其它日志数组之间的第一差值的步骤。
  4. 如权利要求1所述的日志分类规则的生成方法，其中，所述若检测到只存在一种所述第一信息模板，则根据所述第一信息模板确定所述日志集合对应的特征数组，并根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则的步骤包括：
    若检测到只存在一种所述第一信息模板，则确定所述第一信息模板对应的特征模板；
    根据所述特征模板生成所述日志集合对应的特征数组，为所述特征数组中的信息元素添加对应的信息标签，根据添加信息标签后的所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
  5. 如权利要求1所述的日志分类规则的生成方法，其中，所述若检测到只存在一种所述第一信息模板，则根据所述第一信息模板确定所述日志集合对应的特征数组，并根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则的步骤包括：
    若检测到只存在一种所述第一信息模板，则输出提示信息提示用户确认所述第一信息模板是否正确；
    若接收到确定所述第一信息模板正确的确认指令，则根据所述第一信息模板确定所述日志集合对应的特征数组，并根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
  6. 如权利要求1所述的日志分类规则的生成方法，其中，所述当获取到待分类日志，确定所述待分类日志对应的日志类别，并确定各个日志类别对应的日志集合的步骤包括：
    当获取到待分类日志后，将所述待分类日志的每一行日志中的一个信息单元确定为一个元素，以得到所述待分类日志中每一行日志对应的元素数组；
    在所述元素数组中剥离出时间戳元素、设备名称元素和日志类别标识元素后，得到所述待分类日志中每一行日志对应的日志数组；
    将所述设备名称和所述日志类别标识相同的待分类日志确定为属于同一日志类别的日志；
    在同一日志类别的日志对应的日志数组中，将数组长度相同的日志数组组合成日志集合，以得到各个日志类别对应的日志集合。
  7. 如权利要求1至6任一项所述的日志分类规则的生成方法，其中，
    所述若检测到只存在一种所述第一信息模板，则根据所述第一信息模板确定所述日志集合对应的特征数组，并根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则的步骤之后，还包括：
    当获取到新日志后，提取所述新日志中的类别数据，根据所述类别数据确定所述新日志对应的分类规则；
    根据所述新日志对应的分类规则确定所述新日志对应的特征数组，根据所述新日志对应的特征数组处理所述新日志，以对所述新日志进行半结构化处理。
  8. 一种日志分类规则的生成装置，其中，所述日志分类规则的生成装置包括：
    确定模块，用于当获取到待分类日志，确定所述待分类日志对应的日志类别，并确定各个日志类别对应的日志集合；
    读取模块，用于在所述日志集合中读取一个日志数组确定为目标数组；
    计算模块，用于计算所述目标数组与所述日志集合中的其它日志数组之间的第一差值；
    所述确定模块还用于根据所述第一差值确定所述日志集合对应的第一信息模板；
    检测模块，用于检测是否只存在一种所述第一信息模板；
    所述确定模块还用于若检测到只存在一种所述第一信息模板，则根据所述第一信息模板确定所述日志集合对应的特征数组；
    生成模块，用于根据所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
  9. 如权利要求8所述的日志分类规则的生成装置，其中，所述读取模块还用于若检测到存在至少两种所述第一信息模板，则依次读取所述日志集合中除所述目标数组外的其它日志数组作为待计算数组；
    所述计算模块还用于计算所述待计算数组与所述日志集合中除所述待计算数组外的其它日志数组之间的第二差值；
    所述确定模块还用于根据所述第二差值确定所述待计算数组对应的第二信息模板；
    所述计算模块还用于计算所述第二信息模板的出现次数；
    所述检测模块还用于若所述第二信息模板的出现次数大于所述第一信息模板的出现次数，则检测所述待计算数组是否为所述日志集合中的最后一个数组；
    所述确定模块还用于若所述待计算数组为所述日志集合中的最后一个数组，则根据所述第二信息模板确定所述日志集合对应的特征数组。
  10. 如权利要求8所述的日志分类规则的生成装置，其中，所述确定模块还用于若检测到只存在一种所述第一信息模板，则确定所述第一信息模板对应的特征模板；
    所述生成模块还用于根据所述特征模板生成所述日志集合对应的特征数组；
    所述日志分类规则的生成装置还包括：
    添加模块，用于为所述特征数组中的信息元素添加对应的信息标签；
    所述生成模块还用于根据添加信息标签后的所述特征数组和所述日志类别对应的类别数据生成日志的分类规则。
  11. 一种日志分类规则的生成设备，其中，所述日志分类规则的生成设备包括存储器、处理器和存储在所述存储器上并可在所述处理器上运行的日志分类规则的生成程序，所述日志分类规则的生成程序被所述处理器执行时实现如权利要求1至7中任一项所述的日志分类规则的生成方法的步骤。
  12. 一种计算机可读存储介质，其中，所述计算机可读存储介质上存储有日志分类规则的生成程序，所述日志分类规则的生成程序被处理器执行时实现如权利要求1至7中任一项所述的日志分类规则的生成方法的步骤。
PCT/CN2020/087835 2019-06-21 2020-04-29 日志分类规则的生成方法、装置、设备及可读存储介质 Ceased WO2020253399A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910546292.1 2019-06-21
CN201910546292.1A CN110263009B (zh) 2019-06-21 2019-06-21 日志分类规则的生成方法、装置、设备及可读存储介质

Publications (1)

Publication Number Publication Date
WO2020253399A1 true WO2020253399A1 (zh) 2020-12-24

Family

ID=67920589

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/087835 Ceased WO2020253399A1 (zh) 2019-06-21 2020-04-29 日志分类规则的生成方法、装置、设备及可读存储介质

Country Status (2)

Country Link
CN (1) CN110263009B (zh)
WO (1) WO2020253399A1 (zh)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113420032A (zh) * 2021-07-20 2021-09-21 奇安信科技集团股份有限公司 一种日志的分类存储方法及装置
CN113722194A (zh) * 2021-08-02 2021-11-30 中移（杭州）信息技术有限公司 日志统计方法、装置、设备及存储介质
CN113742210A (zh) * 2021-01-18 2021-12-03 北京沃东天骏信息技术有限公司 一种测试数据生成方法、装置、电子设备及存储介质
CN114490296A (zh) * 2022-01-29 2022-05-13 京东科技信息技术有限公司 一种故障诊断的方法和装置
CN114528287A (zh) * 2022-02-22 2022-05-24 广东电网有限责任公司 一种调度日志的校核方法及装置
CN114780510A (zh) * 2022-05-07 2022-07-22 合肥信息工程监理咨询有限公司 一种工作引擎颗粒的自动运行分析系统
CN116359979A (zh) * 2023-01-16 2023-06-30 中石化石油工程技术服务有限公司 一种地震数据道头格式快捷检查方法、装置、设备及介质

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110263009B (zh) * 2019-06-21 2024-01-16 深圳前海微众银行股份有限公司 日志分类规则的生成方法、装置、设备及可读存储介质
WO2021062739A1 (zh) 2019-09-30 2021-04-08 华为技术有限公司 一种分布式系统中消息的同步方法及装置
CN111813757B (zh) * 2020-06-30 2023-02-24 烽火通信科技股份有限公司 特征信息存储方法、装置、设备及可读存储介质
CN112214459A (zh) * 2020-10-12 2021-01-12 成都精灵云科技有限公司 基于事件机制的资源处理流程日志收集系统
CN112367222B (zh) * 2020-10-30 2022-09-27 中国联合网络通信集团有限公司 网络异常检测方法和装置
CN113535955B (zh) * 2021-07-16 2022-10-28 中国工商银行股份有限公司 一种日志快速归类方法及装置
CN114153710B (zh) * 2021-11-12 2023-11-10 苏州浪潮智能科技有限公司 一种基于海量日志的设备状态监控方法、装置及介质
CN117473094B (zh) * 2023-12-27 2024-03-22 南京聚铭网络科技有限公司 日志分类方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101931562A (zh) * 2010-09-29 2010-12-29 杭州华三通信技术有限公司 网络日志的处理方法及设备
CN105550378A (zh) * 2016-02-04 2016-05-04 贵州电网有限责任公司信息中心 一种复杂网络体系下异构安全日志信息的提取与分析方法
US20170075932A1 (en) * 2015-09-16 2017-03-16 Emc Corporation Log storage optimization
CN107992490A (zh) * 2016-10-26 2018-05-04 华为技术有限公司 一种数据处理方法以及数据处理设备
CN110263009A (zh) * 2019-06-21 2019-09-20 深圳前海微众银行股份有限公司 日志分类规则的生成方法、装置、设备及可读存储介质

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8533193B2 (en) * 2010-11-17 2013-09-10 Hewlett-Packard Development Company, L.P. Managing log entries
US9703675B2 (en) * 2014-08-11 2017-07-11 Microsoft Technology Licensing, Llc Structured logging and instrumentation framework
CN106682097B (zh) * 2016-12-01 2020-06-05 北京奇虎科技有限公司 一种处理日志数据的方法和装置
CN106656607A (zh) * 2016-12-27 2017-05-10 上海爱数信息技术股份有限公司 设备日志解析方法、系统及具有该系统的服务器端
CN109543739A (zh) * 2018-11-15 2019-03-29 杭州安恒信息技术股份有限公司 一种日志分类方法、装置、设备及可读存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101931562A (zh) * 2010-09-29 2010-12-29 杭州华三通信技术有限公司 网络日志的处理方法及设备
US20170075932A1 (en) * 2015-09-16 2017-03-16 Emc Corporation Log storage optimization
CN105550378A (zh) * 2016-02-04 2016-05-04 贵州电网有限责任公司信息中心 一种复杂网络体系下异构安全日志信息的提取与分析方法
CN107992490A (zh) * 2016-10-26 2018-05-04 华为技术有限公司 一种数据处理方法以及数据处理设备
CN110263009A (zh) * 2019-06-21 2019-09-20 深圳前海微众银行股份有限公司 日志分类规则的生成方法、装置、设备及可读存储介质

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113742210A (zh) * 2021-01-18 2021-12-03 北京沃东天骏信息技术有限公司 一种测试数据生成方法、装置、电子设备及存储介质
CN113420032A (zh) * 2021-07-20 2021-09-21 奇安信科技集团股份有限公司 一种日志的分类存储方法及装置
CN113722194A (zh) * 2021-08-02 2021-11-30 中移（杭州）信息技术有限公司 日志统计方法、装置、设备及存储介质
CN113722194B (zh) * 2021-08-02 2024-05-24 中移（杭州）信息技术有限公司 日志统计方法、装置、设备及存储介质
CN114490296A (zh) * 2022-01-29 2022-05-13 京东科技信息技术有限公司 一种故障诊断的方法和装置
CN114528287A (zh) * 2022-02-22 2022-05-24 广东电网有限责任公司 一种调度日志的校核方法及装置
CN114780510A (zh) * 2022-05-07 2022-07-22 合肥信息工程监理咨询有限公司 一种工作引擎颗粒的自动运行分析系统
CN114780510B (zh) * 2022-05-07 2024-10-01 合肥信息工程监理咨询有限公司 一种工作引擎颗粒的自动运行分析系统
CN116359979A (zh) * 2023-01-16 2023-06-30 中石化石油工程技术服务有限公司 一种地震数据道头格式快捷检查方法、装置、设备及介质

Also Published As

Publication number Publication date
CN110263009A (zh) 2019-09-20
CN110263009B (zh) 2024-01-16

Similar Documents

Publication Publication Date Title
WO2020253399A1 (zh) 日志分类规则的生成方法、装置、设备及可读存储介质
CN112231407B (zh) PostgreSQL数据库的DDL同步方法、装置、设备和介质
CA2684822C (en) Data transformation based on a technical design document
CN113760891B (zh) 一种数据表的生成方法、装置、设备和存储介质
CN110765101B (zh) 标签的生成方法、装置、计算机可读存储介质及服务器
US10108698B2 (en) Common data repository for improving transactional efficiencies of user interactions with a computing device
US11250035B2 (en) Knowledge graph generating apparatus, method, and non-transitory computer readable storage medium thereof
CN110209684A (zh) 数据库ddl变更操作的跟踪方法、装置、系统及介质
CN112685433A (zh) 元数据更新方法、装置、电子设备及计算机可读存储介质
WO2020207167A1 (zh) 文本分类方法、装置、设备及计算机可读存储介质
CN111553556A (zh) 业务数据分析方法、装置、计算机设备及存储介质
CN106844139A (zh) 一种日志文件分析方法及装置
CN112328631A (zh) 一种生产故障分析方法、装置、电子设备及存储介质
CN110188568A (zh) 机密信息标识方法、装置、设备与计算机可读存储介质
CN103678396B (zh) 一种基于数据模型的数据备份方法和装置
CN110334147A (zh) 一种数据同步方法及装置
CN114218431B (zh) 视频搜索方法、装置、电子设备以及存储介质
CN117033309A (zh) 一种数据转换方法、装置、电子设备及可读存储介质
CN108694172A (zh) 信息输出方法和装置
CN119807141A (zh) 数据标准治理方法、装置、计算机设备及存储介质
CN115774793B (zh) 机构时效性的检测方法、系统、电子设备及存储介质
CN120407645B (zh) 表格数据处理方法、设备、介质和程序产品
CN116610703B (zh) 数据查询方法、装置、计算机设备和存储介质
CN113626867B (zh) 数据权限过滤方法、装置、计算机设备及存储介质
CN116861042B (zh) 基于材料数据库的信息校验方法、装置、设备及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20826788

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20826788

Country of ref document: EP

Kind code of ref document: A1