WO2020140624A1

WO2020140624A1 - Method for extracting data from log, and related device

Info

Publication number: WO2020140624A1
Application number: PCT/CN2019/118038
Authority: WO
Inventors: 陈珍妮
Original assignee: 平安科技（深圳）有限公司
Priority date: 2019-01-04
Filing date: 2019-11-13
Publication date: 2020-07-09
Also published as: CN109783459A

Abstract

A method and apparatus for extracting data from a log, relating to the technical field of artificial intelligence. The method comprises: monitoring log update of the running system (S110); if the log update is monitored, identifying the updated log by means of a neural network model to determine the log type of the updated log (S130); searching for in a configuration file data extraction information corresponding to the log type, the data extraction information indicating a data item where the data is extracted from the log of the log type (S150); and extracting, according to the searched data extraction information, the data corresponding to the data item from the updated log (S170). According to the method, the required data is extracted from the running system in real time, and the efficiency is high.

Description

Specification Title: Method and related equipment for extracting data from logs Technical field

[0001] This application requires the priority of the Chinese patent application CN201910007431.3 submitted on January 4, 2019, with the invention titled "Method, Device and Computer-readable Storage Media for Extracting Data from Logs", hereby incorporated by reference All of its contents are merged here.

[0002] The present application relates to the field of artificial intelligence technology, and in particular, to a method and related equipment for extracting data from logs.

Background technique

[0003] In order to run the system, it is necessary to collect data related to system operation, such as system login user, user login time, successful processing request, failed processing request, response time, processing failure reason, etc., so as to integrate the system Statistical analysis of, for example, system processing efficiency, user preferences, etc.

[0004] In order to obtain data related to the operation of the system, the corresponding data is obtained by searching the corresponding data in the database table of the system. However, the data stored in the database table is not complete. Generally, due to the redundancy of the database, the database table only includes data such as the final result processed by the system. The inventor realized that the collected data depends on the data stored in the database table. If the corresponding data to be collected is not saved in the database table, the data needs to be collected from other channels. The efficiency of data acquisition is low, and the data obtained is not complete.

[0005] It can be seen from the above that the problem of how to effectively obtain data related to system operation has yet to be resolved.

Summary of the invention

technical problem

Solution to the problem

Technical solution

[0006] In order to solve the problem of obtaining data related to system operation in the related art, the present application provides a method and apparatus for extracting data from a log.

[0007] In a first aspect, a method for extracting data from a log includes: performing log update on a running system New monitoring; if the log update is monitored, the updated log is identified through the neural network model to determine the log type of the updated log; the data extraction information search corresponding to the log type in the configuration file is performed, and the data extraction The information indicates a data item for data extraction from the log of the log type; extracting data corresponding to the data item from the updated log according to the found data extraction information.

[0008] In a second aspect, an apparatus for extracting data from a log includes: a monitoring module configured to: perform log update monitoring on a running system; an identification module configured to: if a log update is monitored, then Recognize the updated log through the neural network model to determine the log type of the updated log; the search module is configured to: search for data extraction information corresponding to the log type in the configuration file, and the data extraction information indicates A data item for data extraction from the log of the log type; an extraction module configured to: extract data corresponding to the data item from the updated log according to the found data extraction information.

[0009] In a third aspect, an electronic device includes: a processor; and a memory, where the computer-readable instructions are stored on the memory, and the computer-readable instructions are executed by the processor to implement the following steps:

[0010] Perform log update monitoring on the running system; if a log update is monitored, identify the updated log through a neural network model to determine the log type of the updated log; perform the log type in the configuration file Corresponding to data extraction information search, the data extraction information indicates a data item for data extraction from the log of the log type; extracting the data item corresponding to the data item from the updated log according to the found data extraction information Data

[0011] According to a fourth aspect, a computer non-volatile readable storage medium has stored thereon a computer program, and when the computer program is executed by a processor, the following steps are implemented: performing log update monitoring on the running system; if When the log update is monitored, the updated log is identified through the neural network model to determine the log type of the updated log; the data extraction information corresponding to the log type is searched in the configuration file, and the data extraction information indicates from Data items for data extraction in the log of the log type; extracting data corresponding to the data items from the updated log according to the found data extraction information.

[0012] According to the method of the present application, through log update monitoring, identification of the log type of the updated log, search of data extraction information defined by the log type, and extracting pairs from the updated log according to the data extraction information The data should be extracted from the log to obtain data related to system operation, real-time collection of system operation data, and data integrity is ensured. In addition, the deep learning method is used to identify the type of the local chronicle, which improves the recognition efficiency and accuracy, and ensures the efficiency and real-time performance of data extraction.

[0013] It should be understood that the above general description and the following detailed description are only exemplary and do not limit the present application.

Beneficial effects of invention

Brief description of the drawings

BRIEF DESCRIPTION

[0014] The drawings herein are incorporated into the specification and constitute a part of the specification, show embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0015] Fig. 1 is a block diagram of a server according to an exemplary embodiment;

[0016] FIG. 2 is a flowchart of a method for extracting data from a log according to an exemplary embodiment; [0017] FIG. 3 is a flowchart of step S 130 of the embodiment corresponding to FIG. 2;

[0018] FIG. 4 is a flowchart of steps before step S130 of the embodiment corresponding to FIG. 2;

[0019] FIG. 5 is a flowchart of steps before step S150 of the embodiment corresponding to FIG. 2;

6 is a flowchart of steps after step S170 of the embodiment corresponding to FIG. 2;

[0021] FIG. 7 is a flowchart of step S430 of the embodiment corresponding to FIG. 6;

[0022] Fig. 8 is a block diagram of a device for extracting data from a log according to an exemplary embodiment;

[0023] Fig. 9 is a block diagram of an electronic device according to an exemplary embodiment.

[0024] The above-mentioned drawings have shown clear embodiments of the present application, and will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but by reference. The specific embodiments illustrate the concept of the present application for those skilled in the art.

Invention Example

Embodiments of the invention

[0025] Exemplary embodiments will be described in detail here, examples of which are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. To The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of devices and methods consistent with some aspects of the application as detailed in the appended claims.

[0026] Fig. 1 is a block diagram of a server according to an exemplary embodiment. A server with this hardware structure can be used to perform the method of extracting data from the log of the present application, wherein the system runs on the server to provide services for each terminal of the system, thereby generating logs during the operation of the system, and the server can The generated log is subjected to data extraction according to the method of this application. Of course, the main body of the method of extracting data from the log of the present application is not limited to the server shown in FIG. 1, the main body of the method of the present application may also be a device with logic operation processing capabilities, such as a desktop computer, a laptop computer, and The server cluster, cloud server, etc. composed of multiple servers are not specifically limited herein.

[0027] It should be noted that the server is only an example adapted to the present application, and cannot be considered as providing any limitation on the scope of use of the present application. The server cannot also be interpreted as requiring or having to have one or more components in the exemplary server 200 shown in FIG.

[0028] The hardware structure of the server may vary greatly due to differences in configuration or performance. As shown in FIG. 2, the server 200 includes: a power supply 210, an interface 230, at least one memory 250, and at least one central processor ( CPU, Central Processing Units) 270.

[0029] The power supply 210 is used to provide an operating voltage for each hardware device on the server 200.

The interface 230 includes at least one wired or wireless network interface 231, at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc., for communicating with external devices, such as data with the terminal 100 transmission.

[0031] As a carrier for resource storage, the memory 250 may be a read-only memory, a random access memory, a magnetic disk, or an optical disk. The resources stored on the memory 250 include an operating system 251, application programs 253, and data 255. The storage method may be temporary storage. Or permanent storage. Among them, the operating system 251 is used to manage and control the hardware devices and application programs 253 on the server 200 to implement the calculation and processing of the massive data 255 by the central processor 270, which may be Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM , FreeBSDTM, etc. The application program 253 is a computer program that completes at least one specific job based on the operating system 251, and may include at least one module (not shown in FIG. 2), and each module may separately include a series of computers for the server 200. Readable instructions. Data 255 can be stored on disk Logs etc.

[0032] The central processor 270 may include one or more processors, and is configured to communicate with the memory 250 through a bus for computing and processing the massive data 255 in the memory 250.

[0033] As described in detail above, the server 200 applicable to the present application will complete the method of extracting data from the log by the central processor 270 reading a series of computer-readable instructions stored in the memory 250.

[0034] In an exemplary embodiment, the server 200 may be one or more application specific integrated circuits (Applicati on Specific Integrated Circuit, ASIC for short), digital signal processor, digital signal processing equipment, programmable logic device, field Programming gate arrays, controllers, microcontrollers, microprocessors or other electronic components are implemented to perform the following method of extracting data from the log. Therefore, the implementation of this application is not limited to any specific hardware circuit, software, or a combination of both.

[0035] Fig. 2 is a flow chart showing a method for extracting data from a log according to an exemplary embodiment. The method may be executed by the server shown in FIG. 1, and may include the following steps:

[0036] Step S110: Perform log update monitoring on the running system.

[0037] The running system is, for example, a system that provides services for various program clients, such as a trading system, a valuation system, a fund system, etc. in a financial company, or an application program running on a terminal device

[0038] The system continuously performs logic processing during operation, for example, receiving a request initiated by the client, performing request processing according to the request initiated by the client, and issuing instructions to the client. Of course, depending on the system, the logic performed The treatment is also different. Therefore, during the logical processing of the system, a log is generated according to the logical processing performed, for example, after receiving a request initiated by a client, a log of the received request is generated, or after a request is processed, a log of the result of the request processing is generated, After the client login is successful, the user login log is generated.

[0039] The corresponding log storage unit is configured in the system, so that the logs generated during the operation of the system are stored in the configured log storage unit, so that the log can be updated in the log storage unit Monitoring, that is, whether a new log is stored in the log storage unit for the emergency over the year, the new log is a log newly generated by the running system.

[0040] Step S130: If the log update is monitored, the updated log is identified through the neural network model to determine the log type of the updated log. [0041] In response to different logical operations performed by the system, logs corresponding to different log types are generated, wherein logs with different log types have different formats of logs on the one hand and different data contained in the logs on the other hand. For example, for example, in a system, logs are generated for user login and user request processing results, where the logs generated for user login (called login type logs) are:

[0042] 20180904-11:21: User jenny logged into the system

[0043] The log generated for user request processing success (called the request processing success type log) is:

[0044] 20180904-14:00: User Amy initiated a request for product creation, the processing was successful, and the response time was 2.2 seconds

[0045] For the above-mentioned login type log, which includes the user login time "20180904-11:21" and the login user "jenny", the user login time and the logged-in user included therein are the carried in the login type log data. For the above request processing success type log, including the time "20180 904-14:00" when the user initiated the request, the user "Amy" who initiated the request, the request type "product new", the request processing result "success", and the system response time "2.2 seconds", the time included in the log by the user to initiate the request, the user who initiated the request, the type of request, the request processing result, and the time of the system response are the data carried in the log.

[0046] Since the log format of different log types is different, the updated log can be identified through the neural network model to determine the log type of the updated log, that is, the updated log is identified by deep learning.

[0047] After training, the neural network model performs feature extraction on the updated log, and then predicts the label of the updated log according to the extracted feature, thereby determining the log type of the updated log. Among them, the neural network model used may be a convolutional neural network model, a recursive neural network model, or a recurrent neural network model, which is not specifically limited here.

[0048] Step S150: Search for data extraction information corresponding to the log type in the configuration file, and the data extraction information indicates a data item for data extraction from the log of the log type.

[0049] Wherein, the configuration file includes data extraction information configured for the log corresponding to the log type for which data extraction is required. Therefore, the configuration file may include one or more sets of data extraction information, where one set of data extraction information corresponds to a log type. In the data extraction information corresponding to each log type, data items to be extracted from the log corresponding to the log type are configured. For example, the log in the above example, if you log from the login type in the above example: [0050] 20180904-11:21: User jenny logged into the system

[0051] To extract the login time and the logged-in user, the logged-in time and the logged-in user are data items that need to be extracted, and in the log, the logged-in time “20180904-11:21” corresponds to the logged-in data item The data of the login user "jenny" is the data corresponding to the data item of the login user.

[0052] Step S170: Extract data corresponding to the data item from the updated log according to the found data extraction information.

[0053] The data extraction information corresponding to each log type indicates one or more data items that require data extraction. Therefore, the data corresponding to the data item is extracted from the updated log according to the found data extraction information. Thus, data collection from the log is realized.

[0054] In order to understand the usage of the system, statistics related to the operation of the system need to be counted, such as the amount of system processing success, the amount of system processing failure, the system response time for each request, the number of system login users, and even to analyze user behavior preferences , Need to further count user login time, the type of request initiated by the user, etc. According to the method of the present application, there is no need to search and obtain data from a database storing the operating results of the system, and indirectly obtain data to be collected according to other data sources. And through log update monitoring, identification of the log type of the updated log, search of the data extraction information defined by the log type, and extracting the corresponding data from the updated log according to the data extraction information, so as to achieve the extraction from the log and the system The operation-related data realizes the real-time collection of system operation data. Furthermore, the deep learning method is used to identify the log types, which improves the identification efficiency and accuracy, and further ensures the efficiency and real-time performance of data extraction. In addition, because the system log contains all the information related to the operation of the system, the data extracted from the log guarantees the integrity of the extracted data relative to the method obtained from the database or indirectly.

[0055] In an embodiment, as shown in FIG. 3, step S130 includes:

[0056] Step S131: Construct a feature vector of the updated log.

[0057] The feature vector may be constructed based on the text of the updated log. Because the format of logs of different log types is different, the feature vectors constructed for logs of different log types are also different. The constructed feature vector reflects the characteristics of the updated log.

[0058] For example, for example, the login type log mentioned above, such as

[0059] 20180904-11:21: User jenny logged into the system [0060] and the request processing success type log, such as

[0061] 20180904-14:00: User Amy initiated a request for product creation, the processing was successful, and the response time was 2.2 seconds

[0062] The keywords configured in the logs of the two log types are different. In the log of the log type, XXXXXXXX-XX: XX user XX logged into the system, except for the position occupied by the X symbol. The content of other parts is the same.

[0063] For the request processing success type log: YYYYYYYY-YY: User YY initiated a YYYY request, processing YY, response time YY seconds, except the position occupied by the Y symbol is, the rest of the log type is the same of. It can be seen from the previous two examples that in each type of log, the configured keywords and the location of the keywords are fixed. For example, in the login type log, after the specific login user XX is "Login to the system", so that when constructing the feature vector of a log of a certain log type, the feature vector is constructed according to the keyword in the log and the location of the keyword. That is, the keyword search is performed in the updated log, and the position of the keyword in the updated log is obtained, thereby constructing the feature vector of the updated log.

[0064] Further, before step S131, it further includes segmenting the updated log, and then constructing a feature vector of the updated log according to the encoding corresponding to each word.

[0065] Step S132: Perform classification prediction on the feature vector to obtain a type label corresponding to the updated log.

[0066] Step S133: Determine the log type of the updated log according to the type tag.

[0067] In the neural network model, according to the log generated by the system, configure a corresponding type label for each log type log. Therefore, in step S132, the classification prediction is performed according to the constructed feature vector, that is, the probability of each type label of the feature vector is predicted, and then the probability of predicting each type of label is traversed, and the type label with the maximum probability is used as the The type tag corresponding to the update log. Therefore, the log type of the updated log is determined according to the obtained type log.

[0068] In an embodiment, as shown in FIG. 4, before step S130, the method further includes:

[0069] Step S210: Acquire a plurality of sample logs, and acquire a sample label marked for each sample log.

[0070] Step S220, training the neural network model through several sample logs and corresponding type tags.

[0071] Step S230: When the neural network model converges, the training of the neural network model ends.

[0072] The neural network model predicts the type label of the sample log for each sample log. If the predicted type label is inconsistent with the sample label marked on the sample log, the neural network model is adjusted Type parameters until the predicted type label is consistent with the sample label. Repeat this process for each sample log.

[0073] After a period of training, a prediction accuracy test is performed on the neural network model, that is, several test logs are input into the neural network model, the neural network model predicts the type label of each test log, and the type label of each test log Compared with the type label marked on the test log, if they are consistent, the Bem neural network model predicts the test log accurately, if not, the neural network model predicts the test log incorrectly, so that the neural network is statistically obtained The prediction accuracy of the model (prediction accuracy = number of test logs with accurate prediction/total number of test logs), if the obtained prediction accuracy meets the requirements of the set accuracy, the neural network model after the training converges, thus ending The training of the neural network model, and the neural network model after the training is used to identify the updated log in step S130. By training the neural network model, to improve the recognition accuracy of the update log

[0074] In an embodiment, as shown in FIG. 5, before step S150, it further includes:

[0075] Step S310: Acquire a template log of the same type as the log corresponding to the log to be extracted.

[0076] Step S320: In the template log, replace the data corresponding to the data item with the variable configured for the data item, and obtain data extraction information corresponding to the log type according to the replaced template log configuration.

[0077] Step S330, a configuration file is formed from the data extraction information corresponding to each log type.

[0078] The template log may be any log of the log type. Corresponding to the situation where data needs to be extracted from logs of multiple log types, correspondingly, a template log of each log type is obtained.

[0079] As described above, for different logs of the same log type, the log format is the same, in which there are the same parts, for example, the keywords in the log, and the position of the keywords are the same, but different The part is only a few, such as the data corresponding to the data items that need to be extracted.

[0080] Before configuring the data extraction information, configure variables for each data item that needs to be extracted. In the model log, replace the data corresponding to the data item in the template log with the variable configured for the data item, and define the variable The value is used as output to obtain the data extraction information corresponding to the log type. Therefore, when data is extracted in the log corresponding to the log type according to the data extraction information, the data at the variable position is extracted from the log, which is the data corresponding to the data item.

[0081] For example, for example, two data items of login time and login user need to be extracted from the login type log The corresponding data is configured to obtain data extraction information for the log type:

[0082] Metrics.login.pattern=%timestamp%: SP%usemame% logged into the system

[0083] Metric s. Login. index= timestamp, username

[0084] where timestamp is a variable configured for the data item of login time, and username is a variable configured for the data item of login user. In the first line of the data extraction information, the template log is used to replace the data corresponding to the data item with the variables configured for the data item. That is equivalent to assigning the data corresponding to the data item to the variable configured for the data item. The second line in the data extraction information defines the output variable, that is, the variable corresponding to the data item needs to be extracted as the output variable, so that when data extraction is performed according to the data extraction information, the corresponding data item in the log can be obtained data.

[0085] Since the configuration file may be configured for logs of multiple log types, data extraction information is configured for each log type, and the data extraction information corresponding to each log type constitutes a configuration file.

[0086] Further, in order to facilitate the positioning of the data extraction information corresponding to each log type in the configuration file, the corresponding identification is configured for the data extraction information of each log type, and the identification of the data extraction information and the log type are created Association, so that after identifying the log type of the updated log by identifying in step S130, the data extraction information identifier associated with the log type can be directly searched, so as to quickly find the data extraction information corresponding to the log type.

[0087] In an embodiment, as shown in FIG. 6, after step S170, the method further includes:

[0088] Step S410: Search the data table corresponding to the log type.

[0089] Step S430: Write the extracted data to a data table to store the data.

[0090] For the logs of different log types, the extracted data is different, so that each log type is configured with a corresponding data table for storing the data extracted from the log of the log type. And write the data to the corresponding data table to realize the storage of the extracted data. Therefore, when performing analysis processing, the analysis is performed directly based on the data stored in the data table, and the analysis results are obtained, for example, information such as user login volume, system processing success volume, and system processing failure volume are obtained.

[0091] In an embodiment, as shown in FIG. 7, step S430 includes:

[0092] Step S431: Locate the data field associated with the data item in the data table.

[0093] Step S432: Write the data corresponding to the data item into the table unit configured for the data field. [0094] The data extracted for the log of each log type may be data of one data item, or data of multiple data items. Therefore, for the case where the extracted data is data of multiple data items, a data field is configured for each data item in the data table, and the data item is associated with the data field, thereby writing the extracted data to the data table During data, the data field associated with the data item is located and searched, and then the data of the data item is written into the table unit configured as the data field. Further, in the data table, data is written line by line, that is, after data is written in one line in the data table, the next extracted data is written in the next line of the line, and so on.

[0095] The following is an embodiment of the apparatus of the present application, which may be used to execute an embodiment of the method for extracting data from a log executed by the server 200 of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiment of the present application for extracting data from logs.

[0096] FIG. 8 is a block diagram of an apparatus for extracting data from logs according to an exemplary embodiment. The apparatus may be deployed in the server 200 shown in FIG. 1 and execute any of the above method embodiments. All or part of the method of extracting data from the log. As shown in FIG. 8, the device includes but is not limited to: a monitoring module 110, an identification module 130, a search module 150, and an extraction module 170, wherein: the monitoring module 110 is configured to: perform log update monitoring on the running system. An identification module 130, which is connected to the monitoring module 110, is configured to: if a log update is monitored, identify the updated log through a neural network model to determine the log type of the updated log. A search module 150, which is connected to the identification module 130, is configured to: search for data extraction information corresponding to the log type in the configuration file, and the data extraction information indicates data items to be extracted from the log of the log type. Extraction module 170, which is connected to the search module 150, and is configured to: extract data corresponding to the data item from the updated log according to the found data extraction information.

[0097] For the implementation process of the functions and functions of the various modules in the above device, see the implementation process of the corresponding steps in the above method for extracting data from the log, which will not be repeated here.

[0098] It can be understood that these modules may be implemented by hardware, software, or a combination of both. When implemented in hardware, these modules may be implemented as one or more hardware modules, such as one or more application specific integrated circuits. When implemented in software, these modules may be implemented as one or more computer programs executed on one or more processors, such as the programs stored in the memory 250 executed by the central processor 270 of FIG. 2. [0099] In an embodiment, the recognition module 130 includes: a feature vector construction unit configured to: construct the updated feature vector of the log. The classification prediction unit is configured to: perform classification prediction on the feature vector to obtain the type label corresponding to the updated log. The log type determination unit is configured to: determine the log type of the updated log according to the type label.

[0100] For the implementation process of the functions and functions of each module in the above apparatus, please refer to the implementation process of the corresponding steps in the above method embodiment for details, which will not be repeated here.

[0101] In an embodiment, the device for extracting data from the log further includes the following module, which performs the corresponding steps before the identification module is executed: a sample log acquisition module configured to: acquire a plurality of sample logs, and acquire the same The sample label marked in this log. The training module is configured to: train the neural network model through several sample logs and corresponding type labels. The end of training module is configured to: end the training of the neural network model when the neural network model converges.

[0102] For the implementation process of the functions and functions of the various modules in the above device, see the implementation process of the corresponding steps in the above method embodiments for details, and details are not described herein again.

[0103] In an embodiment, the device for extracting data from the log further includes the following module, which performs the corresponding step before the search module is executed: a template log acquisition module configured to: acquire a log corresponding to the log to be extracted Template logs with the same log type. The data extraction information generating module is configured to: replace the data corresponding to the data item with the variables configured for the data item in the template log, and obtain the data extraction information corresponding to the log type according to the template log configuration after the replacement. The configuration file generation module is configured as follows: the configuration file is composed of data extraction information corresponding to each log type.

[0104] For the implementation process of the functions and functions of the various modules in the above device, see the implementation process of the corresponding steps in the above method embodiment for details, which will not be repeated here.

[0105] In an embodiment, the apparatus for extracting data from the log further includes: a data table search module configured to: perform a search for a data table corresponding to the log type. The data writing module is configured to: write the extracted data into a data table for data storage.

[0106] For the implementation process of the functions and functions of the various modules in the above device, see the implementation process of the corresponding steps in the above method embodiment for details, which will not be repeated here.

[0107] In an embodiment, the data writing module includes: a data field positioning unit configured to: locate the data field associated with the data item in the data table. Write unit, configured to: map data items The data is written to the table cell configured for the data field.

[0108] For the implementation process of the functions and functions of the units in the above device, see the implementation process of the corresponding steps in the above method embodiments for details, and details are not described herein again.

[0109] Optionally, the present application also provides an electronic device, which can be used in the server 200 shown in FIG. 1 to perform all of the methods for extracting data from logs shown in any of the above method embodiments Or some steps. As shown in FIG. 9, the slave electronic device 1000 includes: a processor 1001; and a memory 1002, where the computer readable instructions are stored on the memory 1002, and when the computer readable instructions are executed by the processor 1001, the method of any of the above method implementations is implemented .

[0110] Wherein, when the executable instruction is executed by the processor 1001, the method in any of the above embodiments is implemented. The executable instructions are, for example, computer-readable instructions. When the processor 1001 executes, the processor reads the computer-readable instructions stored in the memory through the communication line/bus 1003 connected to the memory.

[0111] The specific manner in which the processor of the electronic device performs operations in this embodiment has been described in detail in the embodiment regarding the method for extracting data from the log, and will not be elaborated here.

[0112] In an exemplary embodiment, a computer non-volatile readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the slave log in any of the above method embodiments is implemented The method of extracting data. The computer non-volatile readable storage medium includes, for example, a memory 250 of a computer program, and the above instructions can be executed by the central processor 270 of the server 200 to implement the above method of extracting data from the log.

[0113] The specific manner in which the processor performs operations in this embodiment has been described in detail in the embodiment regarding the method of extracting data from the log, and will not be elaborated here.

[0114] It should be understood that the present application is not limited to the precise structure that has been described above and shown in the drawings, and that various modifications and changes can be performed without departing from the scope thereof. The scope of this application is limited only by the appended claims.

Claims

[Claim 1] A method for extracting data from a log, including:

Perform log update monitoring on the running system;

If the log update is monitored, the updated log is identified through the neural network model to determine the log type of the updated log;

Searching for data extraction information corresponding to the log type in the configuration file, the data extraction information indicating data items for data extraction from the log of the log type; based on the found data extraction information from the The data corresponding to the data item is extracted from the update log.

[Claim 2] The method according to claim 1, wherein the identification of the updated log by the neural network model to determine the log type of the updated log includes:

Construct the feature vector of the updated log;

Classify and predict the feature vector to obtain a type tag corresponding to the updated log. Determine the log type of the updated log according to the type tag.

[Claim 3] The method according to claim 1, wherein, before identifying the updated log through the neural network model to determine the log type of the updated log, the method further includes: obtaining a number of sample logs, and obtaining a pair of The sample label marked in each of the sample logs is used to train the neural network model through the several sample logs and the corresponding type tags;

When the neural network model converges, the training of the neural network model ends.

[Claim 4] The method according to claim 1, wherein, before searching the data extraction information corresponding to the log type in the configuration file, the method further includes:

Obtain a template log of the same type as the log corresponding to the log to be extracted; in the template log, replace the data corresponding to the data item with the variable configured for the data item, and according to the replaced Template log configuration to obtain data extraction information corresponding to the log type; The configuration file is constituted by data extraction information corresponding to each of the log types.

[Claim 5] The method according to claim 1, wherein, after extracting the data corresponding to the data item from the updated log according to the found data extraction information, further comprising: performing the log Search for the data table corresponding to the type;

Write the extracted data to the data table to store the data.

[Claim 6] The method according to claim 5, wherein the writing the extracted data into the data table includes:

Positioning the data field associated with the data item in the data table;

Writing data corresponding to the data item into a table cell configured for the data field.

[Claim 7] An apparatus for extracting data from a log, including:

The monitoring module is configured to: monitor the log update of the running system; the identification module is configured to: if the log update is monitored, identify the updated log through the neural network model to determine the log type of the updated log ;

The search module is configured to: perform a search for data extraction information corresponding to the log type in a configuration file, and the data extraction information indicates data items for data extraction from the log of the log type;

The extraction module is configured to: extract data corresponding to the data item from the updated log according to the found data extraction information.

[Claim 8] The apparatus according to Claim 7, wherein the identification module includes:

The feature vector construction unit is configured to: construct the feature vector of the updated log; the classification prediction unit is configured to: perform classification prediction on the feature vector to obtain the type label corresponding to the updated log;

The log type determining unit is configured to: determine the log type of the updated log according to the type tag.

[Claim 9] The device according to claim 7, the device further comprising:

A sample log obtaining module, configured to: obtain a number of sample logs, and obtain a sample label marked for each of the sample logs;

The training module is configured to: enter the sample logs and the corresponding type tags into Perform the training of the neural network model;

The training end module is configured to: end the training of the neural network model when the neural network model converges.

[Claim 10] The apparatus according to claim 7, the apparatus further comprising:

The template log obtaining module is configured to: obtain template logs of the same type as the log corresponding to the log to be extracted;

The data extraction information generating module is configured to: replace the data corresponding to the data item with the variables configured for the data item in the template log, and obtain the log according to the template log configuration after replacement Data extraction information corresponding to the type; a configuration file generation module configured to: constitute the configuration file from the data extraction information corresponding to each of the log types.

[Claim 11] The device according to claim 7, the device further comprising:

The data table search module is configured to: perform a search data writing module of the data table corresponding to the log type, and is configured to: write the extracted data into the data table to store the data .

[Claim 12] The device according to claim 11, the data writing module comprising:

The data field positioning unit is configured to: locate the data field associated with the data item in the data table;

The writing unit is configured to: write data corresponding to the data item into a table unit configured for the data field.

[Claim 13] An electronic device, comprising: a processor; and a memory, computer-readable instructions are stored on the memory, and when the computer-readable instructions are executed by the processor, the following steps are implemented for the running system Carry out log update monitoring;

Searching the data extraction information corresponding to the log type in the configuration file, the data The extraction information indicates a data item for data extraction from the log of the log type; extracting data corresponding to the data item from the updated log according to the found data extraction information.

[Claim 14] The electronic device according to claim 13, wherein in the step of identifying the updated log through the neural network model to determine the log type of the updated log, the processor is configured to :

Construct the feature vector of the updated log;

[Claim 15] The electronic device according to claim 13, wherein the processor is further configured before the step of identifying the updated log through the neural network model to determine the log type of the updated log For:

Acquiring a plurality of sample logs, and acquiring a sample label marked on each of the sample logs to train the neural network model through the several sample logs and the corresponding type labels;

[Claim 16] The electronic device according to claim 13, wherein, before the step of searching for data extraction information corresponding to the log type in the configuration file, the processor is further configured to:

Obtain a template log of the same type as the log corresponding to the log to be extracted; in the template log, replace the data corresponding to the data item with the variable configured for the data item, and according to the replaced Template log configuration to obtain data extraction information corresponding to the log type;

The configuration file is constituted by data extraction information corresponding to each of the log types.

[Claim 17] The electronic device according to claim 13, wherein after the step of extracting data corresponding to the data item from the updated log based on the found data extraction information , The processor is further configured to:

Search the data table corresponding to the log type;

Write the extracted data to the data table to store the data.

[Claim 18] The electronic device according to claim 17, wherein, in the step of writing the extracted data into the data table, the processor is configured to: in the data table Locate the data field associated with the data item in

[Claim 19] A computer non-volatile readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to implement the following steps:

Perform log update monitoring on the running system;

[Claim 20] The computer non-volatile storage medium according to claim 19, wherein in the step of identifying the updated log by the neural network model to determine the log type of the updated log, The processor is configured to:

Construct the feature vector of the updated log;

[Claim 21] The computer non-volatile readable storage medium according to claim 19, wherein before the step of identifying the updated log by the neural network model to determine the log type of the updated log, The processor is also configured to:

Obtaining a number of sample logs, and obtaining the sample label marked for each of the sample logs Training the neural network model through the several sample logs and corresponding type tags;

[Claim 22] The computer non-volatile readable storage medium according to claim 19, wherein before the step of searching for data extraction information corresponding to the log type in the configuration file, the processor It is also configured to:

[Claim 23] The computer non-volatile readable storage medium according to claim 19, wherein, in the extraction log, data corresponding to the data item is extracted from the updated log according to the found data extraction information After the step, the processor is further configured to: perform a search of the data table corresponding to the log type;

Write the extracted data to the data table to store the data.

[Claim 24] The computer nonvolatile readable storage medium according to claim 23, wherein in the step of writing the extracted data into the data table, the processor is configured To locate the data field associated with the data item in the data table;