CN115481167A

CN115481167A - Log processing method and device, storage medium and electronic device

Info

Publication number: CN115481167A
Application number: CN202211049028.5A
Authority: CN
Inventors: 田帅朋
Original assignee: Qingdao Haier Technology Co Ltd; Haier Smart Home Co Ltd; Haier Uplus Intelligent Technology Beijing Co Ltd
Current assignee: Qingdao Haier Technology Co Ltd; Haier Smart Home Co Ltd; Haier Uplus Intelligent Technology Beijing Co Ltd
Priority date: 2022-08-30
Filing date: 2022-08-30
Publication date: 2022-12-16

Abstract

The application discloses a log processing method, a storage medium and an electronic device, which relate to the technical field of smart home/smart home, and the log processing method comprises the following steps: under the condition of acquiring a system log generated by an application system, carrying out primary statistical processing on the system log to obtain basic log data; storing the basic log data into a first database, and storing original log data corresponding to the system log into a second database; in response to the statistical analysis instruction, basic log data corresponding to the statistical analysis instruction are called from the first database to carry out secondary statistical processing, and deep log data are obtained; and calling the original log data matched with the abnormal log data from the second database to perform abnormal investigation processing under the condition that the abnormal log data exists in the deep log data. The method and the device solve the technical problem of low log processing efficiency.

Description

Log processing method and device, storage medium and electronic device

Technical Field

The present disclosure relates to the field of computers, and in particular, to a log processing method, a storage medium, and an electronic apparatus.

Background

With the increase of users, the system log amount also increases in a geometric exponential manner, but the common log processing method in the related art does not well cope with the phenomenon of the increase of the log amount, or the common log processing method in the related art has a problem of low processing efficiency in a high log amount scene. Namely, the related art has a technical problem that log processing efficiency is low.

Disclosure of Invention

The embodiment of the application provides a log processing method and device, a storage medium and an electronic device, and aims to at least solve the technical problem of low log processing efficiency.

According to an aspect of an embodiment of the present application, there is provided a log processing method, including:

under the condition of acquiring a system log generated by an application system, carrying out primary statistical processing on the system log to obtain basic log data;

storing the basic log data into a first database, and storing original log data corresponding to the system log into a second database;

in response to a statistical analysis instruction, calling basic log data corresponding to the statistical analysis instruction from the first database to perform secondary statistical processing to obtain deep log data;

and calling the original log data matched with the abnormal log data from the second database to perform abnormal investigation processing when the abnormal log data exists in the deep log data.

According to another aspect of the embodiments of the present application, there is also provided a log processing apparatus, including: a log processing apparatus, comprising:

the first processing unit is used for carrying out primary statistical processing on the system log under the condition of acquiring the system log generated by the application system to obtain basic log data;

the first storage unit is used for storing the basic log data into a first database and storing original log data corresponding to the system log into a second database;

the second processing unit is used for responding to a statistical analysis instruction, calling basic log data corresponding to the statistical analysis instruction from the first database to perform secondary statistical processing, and obtaining deep log data;

and a third processing unit, configured to, when there is abnormal log data in the deep log data, call, from the second database, original log data matched with the abnormal log data to perform exception checking processing.

As an optional solution, the first processing unit includes: the first extraction module is used for extracting the content of the system log under the condition of acquiring the system log to obtain the content of the system log; and the first statistical module is used for counting the system log according to the system log content to obtain the basic log data.

As an optional solution, the first statistical module includes: the first processing submodule is used for carrying out classification processing on the system logs according to the basic attribute types to which the system log contents belong to obtain classification results; and the first statistical submodule is used for counting the classification result to obtain basic log data under each basic attribute type.

As an optional solution, the third processing unit includes: a first obtaining module, configured to obtain, when the abnormal log data exists in the deep log data, a log data identifier corresponding to the abnormal log data; the first indexing module is used for indexing the original log data stored in the second database by using the log data identifier to obtain original log data matched with the log data identifier; and the first calling module is used for calling the original log data matched with the log data identifier to perform exception troubleshooting.

As an optional solution, the first invoking module includes: a first obtaining sub-module, configured to obtain multiple process logs recorded in original log data with matching log data identifiers, where the process logs are logs generated when the application system runs to each process; a first determining sub-module, configured to determine, from the multiple process logs, an abnormal process log that matches the abnormal log data; and the second determining submodule is used for determining the process corresponding to the abnormal process log in the application system as an abnormal process.

As an optional scheme, before the first processing unit performs a statistical process on the system log to obtain basic log data, the method includes at least one of the following steps; a first obtaining unit, configured to obtain the system log uploaded by the application system through a log gateway; and the second acquisition unit is used for acquiring the system log written into the log directory by the application system according to a preset rule.

As an optional scheme, after the storing the basic log data into a first database and storing the original log data corresponding to the system log into a second database, the first processing unit includes: the first display unit is used for displaying a first identifier of the first database and a second identifier of the second database on a system interface; the second display unit is used for responding to a first display instruction triggered by the first identifier and displaying the basic log data; or responding to a second display instruction triggered by the second identifier, and displaying the original log data.

According to yet another aspect of embodiments herein, there is provided a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the log processing method as above.

According to another aspect of the embodiments of the present application, there is also provided an electronic apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the log processing method through the computer program.

In the embodiment of the application, under the condition that a system log generated by an application system is obtained, the system log is subjected to primary statistical processing to obtain basic log data; storing the basic log data into a first database, and storing original log data corresponding to the system log into a second database; in response to a statistical analysis instruction, calling basic log data corresponding to the statistical analysis instruction from the first database to perform secondary statistical processing to obtain deep log data; under the condition that abnormal log data exist in the deep log data, the original log data matched with the abnormal log data are called from the second database to perform abnormal investigation, and through the technical means of primary processing and secondary processing of the log data, the purposes of efficiently completing the abnormal investigation and reducing the influence degree of abnormal factors on the log processing efficiency when the log data are abnormal are achieved, so that the technical effect of improving the log processing efficiency is realized, and the technical problem of low log processing efficiency is solved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.

Fig. 1 is a schematic diagram of a hardware environment of an interaction method of an intelligent device according to an embodiment of the present application;

FIG. 2 is a schematic flow chart diagram illustrating an alternative log processing method according to an embodiment of the present application;

FIG. 3 is a schematic diagram of an example of another alternative log processing method according to an embodiment of the application;

FIG. 4 is a schematic diagram of an example of another alternative log processing method according to an embodiment of the application;

FIG. 5 is a schematic diagram of an example of another alternative log processing method according to an embodiment of the application;

FIG. 6 is a schematic diagram of an example of another alternative log processing method according to an embodiment of the application;

FIG. 7 is a schematic diagram of an example of another alternative log processing method according to an embodiment of the application;

FIG. 8 is a schematic diagram of an example of another alternative log processing method according to an embodiment of the application;

FIG. 9 is a schematic diagram of an example of another alternative log processing method according to an embodiment of the application;

FIG. 10 is a schematic diagram of an alternative log processing method apparatus according to an embodiment of the application;

fig. 11 is a schematic structural diagram of an alternative electronic device according to an embodiment of the present application.

Detailed Description

In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the accompanying drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

According to one aspect of the embodiment of the application, an interaction method of intelligent household equipment is provided. The interaction method of the intelligent Home equipment is widely applied to full-house intelligent digital control application scenes such as intelligent homes (Smart Home), intelligent homes, intelligent Home equipment ecology, intelligent house (intelligence House) ecology and the like. Optionally, in this embodiment, the interaction method of the smart home device may be applied to a hardware environment formed by the terminal device 102 and the server 104 as shown in fig. 1. As shown in fig. 1, the server 104 is connected to the terminal device 102 through a network, and may be configured to provide a service (e.g., an application service) for the terminal or a client installed on the terminal, provide a database on or independent of the server for providing a data storage service for the server 104, and configure a cloud computing and/or edge computing service on or independent of the server for providing a data operation service for the server 104.

The network may include, but is not limited to, at least one of: wired networks, wireless networks. The wired network may include, but is not limited to, at least one of: wide area networks, metropolitan area networks, local area networks, which may include, but are not limited to, at least one of the following: WIFI (Wireless Fidelity), bluetooth. Terminal equipment 102 can be and not be limited to PC, the cell-phone, the panel computer, intelligent air conditioner, intelligent cigarette machine, intelligent refrigerator, intelligent oven, intelligent kitchen range, intelligent washing machine, intelligent water heater, intelligent washing equipment, intelligent dish washer, intelligent projection equipment, the intelligent TV, intelligent clothes hanger, intelligent (window) curtain, intelligence audio-visual, smart jack, intelligent stereo set, intelligent audio amplifier, intelligent new trend equipment, intelligent kitchen guarding's equipment, intelligent bathroom equipment, the intelligence robot of sweeping the floor, the intelligence robot of wiping the window, intelligence robot of mopping the floor, intelligent air purification equipment, intelligent steam ager, intelligent microwave oven, intelligent kitchen guarding, intelligent clarifier, intelligent water dispenser, intelligent lock etc..

Optionally, as an optional implementation manner, as shown in fig. 2, the log processing method includes:

s202, under the condition that the system log generated by the application system is obtained, carrying out primary statistical processing on the system log to obtain basic log data;

s204, storing the basic log data into a first database, and storing original log data corresponding to the system log into a second database;

s206, responding to the statistical analysis instruction, calling basic log data corresponding to the statistical analysis instruction from the first database to perform secondary statistical processing to obtain deep log data;

and S208, under the condition that abnormal log data exist in the deep log data, calling the original log data matched with the abnormal log data from the second database to perform abnormal investigation processing.

Optionally, in this embodiment, the application system is a third-party application system, and a log meeting the full link log specification is generated; the log gateway is a part of a full link system and is an autonomously developed system; the Flume server is a log acquisition system; the Kafak cluster is used for storing logs and providing log data for a real-time program; the Flink real-time program is a log analysis program based on the Flink, and is an independently developed system; the Elasticissearch is a distributed, high-expansion and high-real-time search and data analysis engine; mysql is a database.

Optionally, in this embodiment, the log processing method may be applied, but not limited, to a company application system, as company services expand, systems are also continuously added, and dependence and cooperation among the systems are more and more. When problems occur, the more systems are involved, the greater the troubleshooting difficulty is. In terms of product operation, in order to get more detailed data, each action of the user is known, and the detailed data is also needed for support. As users increase, the amount of background logs also increases in a geometric exponential manner, which also puts higher demands on the analysis system. The prior art does not have a technical means for the background log with a large amount, and further has the technical problems that when log data is abnormal, the difficulty of abnormal troubleshooting is high, the cost is high, and the log processing efficiency is low.

The system realizes high availability and high concurrency of the system based on Kafka (a high-throughput distributed publish-subscribe message system). In order to deal with massive data, the system realizes real-time data operation based on Flink (a distributed processing framework and a distributed processing engine with high throughput, low delay and high performance). Aiming at the original log, the system stores and retrieves based on the Elasticissearch (distributed, high-expansion and high-real-time search and data analysis engine). The efficiency of secondary processing of the log data by a subsequent full-link system is improved through primary processing, the original data are provided and placed in the second database through secondary processing, exception troubleshooting can be directly and efficiently completed when an exception occurs, and the technical problem that the log processing efficiency is low in the prior art is solved.

Optionally, in this embodiment, the first statistical processing is a process of obtaining basic log data by extracting and classifying according to content of original log data corresponding to the system log, and the second statistical processing is a process of calling a statistical analysis instruction and obtaining deep log data from the basic log data by analysis; the first database is a database stored with basic log data, and the second database is a database stored with original log data; the exception troubleshooting is a process of troubleshooting an exception problem occurring by a user through log data, and may be, but not limited to, troubleshooting a cause of the exception problem, in which process, and after which operation instruction is executed.

Optionally, in this embodiment, the system log may include, but is not limited to, basic information such as a timestamp, a log level, and parameter information, where the log level may be generally divided into four levels: ERROR is a serious ERROR, mainly a program ERROR, including information causing the program to terminate operation, WARN is a general warning and cannot influence the whole operation of the program, INFO is information to be displayed generally, such as information of startup, shutdown and mode switching, etc., DEBUG is debugging information of the program, and the lowest level data is the most.

It should be noted that, in the case of acquiring a system log generated by an application system, the system log is subjected to a statistical processing to obtain basic log data; storing the basic log data into a first database, and storing original log data corresponding to the system log into a second database; in response to the statistical analysis instruction, calling basic log data corresponding to the statistical analysis instruction from the first database to perform secondary statistical processing to obtain deep log data; and calling the original log data matched with the abnormal log data from the second database to perform abnormal investigation processing under the condition that the abnormal log data exists in the deep log data. The beneficial effect of improving the abnormal troubleshooting efficiency is achieved.

For further example, as shown in fig. 3, optionally, a system log 302 generated during an operation process of the intelligent air conditioner is obtained, original data 304 corresponding to the system log 302 is stored in a second database 306, the original data 304 of the system log is divided into basic log data of a wind power control module, basic log data of a temperature sensing module, basic log data of a heat dissipation module, and the like according to contents of the system log, and the basic log data is stored in a first database 308, when a problem occurs in a temperature sensing module of the air conditioner, a research and development worker obtains the basic log data of the temperature sensing module from the first database 308, calls a statistical analysis instruction, obtains deep log data 310 through the basic log data, the deep log data 310 includes logs at an ERROR level when temperature sensing is abnormal, and obtains log data at levels of WARN, INFO, DEBUG, and the like corresponding to a component corresponding to a device ID from the original log data through indexing, thereby troubleshooting the cause of the abnormal problem occurring in which process an operation instruction is executed.

According to the embodiment provided by the application, under the condition that the system log generated by the application system is obtained, the system log is subjected to one-time statistical processing to obtain basic log data; storing the basic log data into a first database, and storing original log data corresponding to the system log into a second database; in response to the statistical analysis instruction, calling basic log data corresponding to the statistical analysis instruction from the first database to perform secondary statistical processing to obtain deep log data; under the condition that abnormal log data exist in deep log data, original log data matched with the abnormal log data are called from a second database to perform abnormal troubleshooting, and through the technical means of primary processing and secondary processing of the log data, the purposes of efficiently finishing the abnormal troubleshooting and reducing the influence degree of abnormal factors on the log processing efficiency when the log data are abnormal are achieved, so that the technical effect of improving the log processing efficiency is achieved, and the technical problem of low log processing efficiency is solved.

As an optional scheme, in the case of acquiring a system log generated by an application system, performing a statistical process on the system log to obtain basic log data, including:

s1, under the condition that a system log is obtained, extracting the content of the system log to obtain the content of the system log;

and S2, counting the system logs according to the system log content to obtain basic log data.

Optionally, in this embodiment, the content extraction is a technical means for reducing the dimension, and may be, but not limited to, extracting content such as a timestamp, a log level, component information, a device ID, and parameter information from the raw log data of the system log. The sequential statistics may be, but not limited to, statistical processing is performed according to preset key fields or prefixes, the basic log data is divided into various basic attribute types and various different levels, and analysis processing may be, but not limited to, performed by using an iterative calculation means.

It should be noted that, in the case of obtaining the system log, the content of the system log is extracted to obtain the content of the system log; and counting the system logs according to the system log content to obtain basic log data. The beneficial effect of improving the comprehensiveness of the analysis log data is achieved

For further example, optionally, for example, when a developer writes a program of the intelligent air conditioner, a developer adds a keyword temparature to a log of a TEMPERATURE sensing module in a log system of the intelligent air conditioner, adds a WIND keyword to a log of a WIND power control module, adds a radar keyword to a log of a heat dissipation module, adds different log grades and parameter information according to the importance degrees of log information generated in different scenes, obtains the log grades in the system log generated in the operation process of the intelligent air conditioner, filters log data at the DEBUG and INFO grades, circularly judges the key log information in each log, and classifies the key log information to obtain the basic log data of the TEMPERATURE sensing module under the TEMPERATURE sensing type, the basic log data of the WIND power control module under the WIND power control type, and the basic log data of the heat dissipation module under the heat dissipation type, wherein the basic log data of each basic type contains log data at the ERROR and WARN grades.

According to the embodiment provided by the application, under the condition that the system log is obtained, the content of the system log is extracted to obtain the content of the system log; the system logs are counted according to the system log content to obtain basic log data, and therefore the purpose of improving the log counting and analyzing efficiency is achieved, and the technical effect of efficiently processing data is achieved.

As an optional scheme, counting the system logs according to the system log content to obtain basic log data, including:

s1, classifying the system logs according to basic attribute types to which the system log contents belong to obtain classification results;

and S2, counting the classification result to obtain basic log data under each basic attribute type.

According to the embodiment provided by the application, the system logs are classified according to the basic attribute types to which the system log contents belong, and classification results are obtained; and counting the classification result to obtain basic log data under each basic attribute type, so that the purpose of refining the log data by combining classification and statistics is achieved, and the technical effect of improving the regularity of the log data is realized.

As an optional scheme, in the case that abnormal log data exists in the deep log data, the original log data matched with the abnormal log data is called from the second database to perform exception troubleshooting, including:

s1, under the condition that abnormal log data exist in deep log data, obtaining a log data identifier corresponding to the abnormal log data;

s2, indexing original log data stored in a second database by using the log data identifier to obtain original log data matched with the log data identifier;

and S3, calling the original log data matched with the log data identifier to perform exception troubleshooting.

Optionally, in this embodiment, the process log is a log generated when the application system runs to each process, and the abnormal process log is a log generated by a process performed when an abnormality occurs.

To further illustrate, alternatively, for example, as shown in fig. 4, the developer calls a statistical analysis instruction from the first database 402, and reads the ERROR level log, for example, the content of the ERROR level log is 2021. The threshold of the time range of the occurrence of the problem is preset to be 3 seconds, the keyword is an abnormal logging system, log data in the time range is read from the second database 404, abnormal flow logs 406 which are matched with abnormal log data and comprise log levels such as ERROR, WARN, INFO and DEBUG and correspond to the abnormal log data are matched for the abnormal flow logs by the keyword, and research and development personnel can find out the reason of the occurrence of the problem according to the log information in the flow in the time period so as to process the abnormal flow logs.

According to the embodiment provided by the application, under the condition that abnormal log data exist in the deep log data, the log data identification corresponding to the abnormal log data is obtained; indexing the original log data stored in the second database by using the log data identifier to obtain original log data matched with the log data identifier; the original log data matched with the log data identification is called to perform exception troubleshooting, and time and key fields can be combined for troubleshooting, so that the technical effect of improving the accuracy of troubleshooting is achieved.

As an optional scheme, invoking the original log data with the log data identifier matching to perform exception troubleshooting, including:

s1, acquiring a plurality of process logs recorded in original log data matched with log data identifiers, wherein the process logs are logs generated when an application system runs to each process;

s2, determining an abnormal flow log matched with the abnormal log data from the plurality of flow logs;

and S3, determining the process corresponding to the abnormal process log in the application system as the abnormal process.

Optionally, in this embodiment, the checking content mainly refers to the specific flow in which the abnormal problem occurs, and may be, but not limited to, the time, the keyword, and the like when the abnormal problem occurs, and this is not unnecessarily limited herein.

It should be noted that, a plurality of process logs recorded in the original log data matched with the log data identifier are obtained, where the process logs are logs generated when the application system runs to each process; determining an abnormal flow log matched with the abnormal log data from the plurality of flow logs; and determining the flow corresponding to the abnormal flow log in the application system as the abnormal flow. The method has the beneficial effect of improving the comprehensiveness of the investigation.

According to the embodiment provided by the application, a plurality of process logs recorded in original log data matched with the log data identification are obtained, wherein the process logs are logs generated when the application system runs to each process; determining an abnormal flow log matched with the abnormal log data from the plurality of flow logs; the flow corresponding to the abnormal flow log in the application system is determined as the abnormal flow, so that the aim of improving the troubleshooting diversity is fulfilled, and the technical effect of improving the efficiency of abnormal troubleshooting is achieved.

As an optional solution, before performing a statistical processing on the system log to obtain basic log data, the method includes at least one of the following steps;

s1, acquiring a system log uploaded by an application system through a log gateway;

and S2, acquiring a system log written into a log directory by the application system according to a preset rule.

Optionally, in this embodiment, after the application system generates the log, the application system may, but is not limited to, upload the log by calling RestAPI of the log gateway, or write the log in a designated directory according to rules and collect the log by using flash.

It should be noted that, the system log uploaded by the application system through the log gateway is obtained; the method has the advantages that the system logs in the log catalog are written into the acquisition application system according to the preset rule, and the diversity of acquisition means is improved.

For further example, optionally, as shown in fig. 5, the application system 502 uploads the system log through the log gateway 504, or writes the system log into the flash 508 in the directory 506 according to rules, reads data in the kafka 508, and writes the original data in the flash 510 into the second database elastic search512 through one-time processing, so as to obtain the original log. And then, the data in the flink 510 is written into the first database mysql 514 through secondary processing, a basic log is stored, and finally, when abnormal inspection is carried out, the content in the original log is indexed through the content corresponding to the basic log, so that the abnormal flow can be eliminated.

According to the embodiment provided by the application, the system log uploaded by an application system through a log gateway is obtained; the system log in the log catalog is written into by the acquired application system according to the preset rule, so that the aim of improving the diversity of data acquisition means is fulfilled, and the technical effect of improving the data processing efficiency is realized.

As an optional solution, after storing the basic log data into the first database and storing the original log data corresponding to the system log into the second database, the method includes:

s1, displaying a first identifier of a first database and a second identifier of a second database on a system interface;

s2, responding to a first display instruction triggered by the first identifier, and displaying basic log data; or, in response to a second presentation instruction triggered by the second identifier, displaying the original log data.

Optionally, in this embodiment, the full link system may perform statistical analysis on data in different dimensions and different time periods, provide a UI for data display and original log display, and display basic log data and identification log data through identification.

It should be noted that, the first identifier of the first database and the second identifier of the second database are displayed on the system interface; displaying the basic log data in response to a first display instruction triggered by the first identifier; or, the original log data is displayed in response to a second display instruction triggered by the second identifier, so that the beneficial effect of high-efficiency display is achieved.

For further illustration, as shown in fig. 6, the content displayed on the interface is composed of a UI 602, a full link system 604, an elastic search 608, and a Mysql610, which improves the simplicity and intuitiveness of the page display.

According to the embodiment provided by the application, a first identifier of a first database and a second identifier of a second database are displayed on a system interface; responding to a first display instruction triggered by the first identifier, and displaying basic log data; or, the original log data are displayed in response to a second display instruction triggered by a second identifier, so that the aim of directly displaying through the identifier is fulfilled, and the technical effect of high-efficiency display is achieved.

For convenience of understanding, the log processing method is applied to a specific log processing scene:

optionally, as shown in fig. 7, after the application system generates the log, the application system may upload the log by calling RestAPI of the log gateway, or write the log in a designated directory according to a rule, and collect the log by using flash. For example, as shown in fig. 8, after receiving the log, the log gateway writes the log content into the corresponding kafka. The logs collected by Flume are also written into Kafka.

Alternatively, for example, as shown in fig. 8, the Flink real-time program may subscribe to topic in kafka, and after a new log is written into kafka, the Flink real-time program may read the log, parse the log, and perform basic operations on the log.

Optionally, as shown in fig. 9, the Flink writes the parsed data into Mysql to provide basic data for the full link system, and after the full link takes the data, the full link may perform higher-dimensional data statistical analysis. Meanwhile, the flash real-time program writes the original log into the elastic search and provides the original log for the full link system, so that the user can troubleshoot the problem.

It is understood that in the specific implementation of the present application, related data such as user information, when the above embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of related data need to comply with related laws and regulations and standards of related countries and regions.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present application is not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.

According to another aspect of the embodiments of the present application, there is also provided a log processing apparatus for implementing the log processing method. As shown in fig. 10, the apparatus includes:

the first processing unit 1002 is configured to perform primary statistical processing on a system log generated by an application system to obtain basic log data when the system log is obtained;

a first storage unit 1004, configured to store the basic log data in a first database, and store the original log data corresponding to the system log in a second database;

the second processing unit 1006, configured to respond to the statistical analysis instruction, call, from the first database, basic log data corresponding to the statistical analysis instruction to perform secondary statistical processing, so as to obtain deep log data;

and a third processing unit 1008, configured to, in a case where there is abnormal log data in the deep log data, call, from the second database, the original log data matched with the abnormal log data to perform exception checking processing.

Optionally, in this embodiment, the application system is a third-party application system, and will produce a log that meets the full link log specification; the log gateway is a part of a full link system and is an autonomously developed system; the Flume server is a log acquisition system; the Kafak cluster is used for storing logs and providing log data for a real-time program; the Flink real-time program is a log analysis program based on Flink, and is an independently developed system; the Elasticissearch is a distributed, high-expansion and high-real-time search and data analysis engine; mysql is a database.

Optionally, in this embodiment, the log processing apparatus may be applied to, but not limited to, a company application system, and as company services expand, systems are also continuously added, and dependence and cooperation among the systems are more and more. When problems occur, the more systems are involved, the greater the difficulty of troubleshooting is. In terms of product operation, in order to get more detailed data, knowing every action of the user also requires detailed data to support. As users increase, the amount of background logs also increases in a geometric exponential manner, which also puts higher demands on the analysis system. The prior art does not have a technical means for the background log with a large amount, and further has the technical problems that when log data is abnormal, the difficulty of abnormal troubleshooting is high, the cost is high, and the log processing efficiency is low.

The system realizes high availability and high concurrency of the system based on Kafka (a high-throughput distributed publish-subscribe message system). In order to deal with massive data, the system realizes real-time data operation based on Flink (a distributed processing framework and a distributed processing engine with high throughput, low delay and high performance). Aiming at the original log, the system stores and retrieves based on the Elasticissearch (distributed, high-expansion and high-real-time search and data analysis engine). The efficiency of secondary processing of the log data by a subsequent full-link system is improved through primary processing, original data are provided and placed in the second database through secondary processing, exception troubleshooting can be directly and efficiently completed when an exception occurs, and the technical problem that log processing efficiency is low in the prior art is solved.

Optionally, in this embodiment, the first statistical processing is a process of obtaining basic log data by extracting and classifying according to content of original log data corresponding to the system log, and the second statistical processing is a process of calling a statistical analysis instruction and obtaining deep log data from the basic log data by analysis; the first database is a database stored with basic log data, and the second database is a database stored with original log data; the exception troubleshooting process is a process of troubleshooting an abnormal problem occurring by a user through log data, and may be, but not limited to, a cause of occurrence of the abnormal problem, in which process, and after which operation instruction is executed.

Optionally, in this embodiment, the system log may include, but is not limited to, basic information such as a timestamp, a log level, and parameter information, where the log level may be generally divided into four levels: ERROR is a serious ERROR, mainly a program ERROR, including information causing program termination, WARN is a general warning, which does not affect the whole program operation, INFO is information to be displayed generally, such as information of startup, shutdown, mode switching, etc., DEBUG is program debugging information, and the lowest level data is the most.

According to the embodiment provided by the application, under the condition that the system log generated by the application system is obtained, the system log is subjected to one-time statistical processing to obtain basic log data; storing the basic log data into a first database, and storing original log data corresponding to the system log into a second database; in response to the statistical analysis instruction, basic log data corresponding to the statistical analysis instruction are called from the first database to carry out secondary statistical processing, and deep log data are obtained; under the condition that abnormal log data exist in deep log data, original log data matched with the abnormal log data are called from a second database to perform abnormal troubleshooting, and through the technical means of primary processing and secondary processing of the log data, the purposes of efficiently finishing the abnormal troubleshooting and reducing the influence degree of abnormal factors on the log processing efficiency when the log data are abnormal are achieved, so that the technical effect of improving the log processing efficiency is achieved, and the technical problem of low log processing efficiency is solved.

For a specific embodiment, reference may be made to the example shown in the log processing apparatus, and details in this example are not described herein again.

As an optional solution, the first processing unit includes: the first extraction module is used for extracting the content of the system log under the condition of obtaining the system log to obtain the content of the system log; and the first statistical module is used for carrying out statistics on the system log according to the system log content to obtain basic log data.

For a specific embodiment, reference may be made to the example shown in the log processing method, which is not described herein again in this example.

As an optional solution, the first statistical module includes: the first processing submodule is used for carrying out classification processing on the system logs according to the basic attribute types of the system logs to obtain classification results; and the first statistical submodule is used for carrying out statistics on the classification result to obtain basic log data under each basic attribute type.

As an optional solution, the third processing unit includes: the first acquisition module is used for acquiring a log data identifier corresponding to abnormal log data under the condition that the abnormal log data exists in the deep log data; the first indexing module is used for indexing the original log data stored in the second database by using the log data identifier to obtain the original log data matched with the log data identifier; and the first calling module is used for calling the original log data matched with the log data identifier to perform exception troubleshooting.

As an optional solution, the first invoking module includes: the first obtaining sub-module is used for obtaining a plurality of process logs recorded in original log data matched with the log data identification, wherein the process logs are logs generated when the application system runs to each process; the first determining submodule is used for determining an abnormal flow log matched with the abnormal log data from the plurality of flow logs; and the second determining submodule is used for determining the process corresponding to the abnormal process log in the application system as the abnormal process.

As an optional scheme, before the first processing unit performs a statistical process on the system log to obtain basic log data, the method includes at least one of the following steps; the system comprises a first acquisition unit, a second acquisition unit and a third acquisition unit, wherein the first acquisition unit is used for acquiring a system log uploaded by an application system through a log gateway; and the second acquisition unit is used for acquiring the system log written into the log directory by the application system according to a preset rule.

For a specific embodiment, reference may be made to the example shown in the log processing method, which is not described herein again.

As an optional scheme, after storing the basic log data in a first database and storing the original log data corresponding to the system log in a second database, the first processing unit includes: the first display unit is used for displaying a first identifier of the first database and a second identifier of the second database on the system interface; the second display unit is used for responding to a first display instruction triggered by the first identifier and displaying the basic log data; or, in response to a second presentation instruction triggered by the second identifier, displaying the original log data.

According to another aspect of the embodiments of the present application, there is also provided an electronic device for implementing the log processing method, as shown in fig. 11, the electronic device includes a memory 1102 and a processor 1104, the memory 1102 stores therein a computer program, and the processor 1104 is configured to execute the steps in any one of the method embodiments by the computer program.

Optionally, in this embodiment, the electronic apparatus may be located in at least one network device of a plurality of network devices of a computer network.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:

s1, under the condition that a system log generated by an application system is obtained, carrying out primary statistical processing on the system log to obtain basic log data;

s2, storing the basic log data into a first database, and storing original log data corresponding to the system log into a second database;

s3, responding to the statistical analysis instruction, calling basic log data corresponding to the statistical analysis instruction from the first database to perform secondary statistical processing, and obtaining deep log data;

and S4, calling the original log data matched with the abnormal log data from the second database to perform abnormal investigation processing under the condition that the abnormal log data exists in the deep log data.

Alternatively, it can be understood by those skilled in the art that the structure shown in fig. 11 is only an illustration, and the electronic device may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, and a Mobile Internet Device (MID), PAD, etc. Fig. 11 is a diagram illustrating a structure of the electronic device. For example, the electronic device may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 11, or have a different configuration than shown in FIG. 11.

The memory 1102 may be configured to store software programs and modules, such as program instructions/modules corresponding to the log processing method and apparatus in the embodiment of the present application, and the processor 1104 executes various functional applications and data processing by running the software programs and modules stored in the memory 1102, so as to implement the log processing method described above. The memory 1102 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 1102 can further include memory located remotely from the processor 1104 and such remote memory can be coupled to the terminal via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory 1102 may be specifically, but not limited to, used for storing information such as system logs, raw log data, basic log data, and the like. As an example, as shown in fig. 11, the memory 1102 may include, but is not limited to, a first processing unit 1002, a first storage unit 1004, a second processing unit 1006, and a third processing unit 1008 in the log processing apparatus. In addition, the log processing device may further include, but is not limited to, other module units in the log processing device, which is not described in detail in this example.

Optionally, the transmitting device 1106 is used for receiving or transmitting data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 1106 includes a Network adapter (NIC) that can be connected to a router via a Network cable to communicate with the internet or a local area Network. In one example, the transmission device 1106 is a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.

In addition, the electronic device further includes: a display 1108 for displaying the information such as the system log, the basic log data, and the original log data; and a connection bus 1110 for connecting the respective module components in the above-described electronic apparatus.

In other embodiments, the terminal device or the server may be a node in a distributed system, where the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting a plurality of nodes through a network communication. The nodes may form a Peer-To-Peer (P2P) network, and any type of computing device, such as a server, a terminal, and other electronic devices, may become a node in the blockchain system by joining the Peer-To-Peer network.

According to an aspect of the application, there is provided a computer program product comprising a computer program/instructions containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section, and/or installed from a removable medium. When executed by the central processing unit, the computer program performs various functions provided by the embodiments of the present application.

The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.

It should be noted that the computer system of the electronic device is only an example, and should not bring any limitation to the function and the use range of the embodiment of the present application.

The computer system includes a Central Processing Unit (CPU) that can perform various appropriate actions and processes according to a program stored in a Read-Only Memory (ROM) or a program loaded from a storage section into a Random Access Memory (RAM). In the random access memory, various programs and data necessary for the operation of the system are also stored. The central processor, the read-only memory and the random access memory are connected with each other through a bus. An Input/Output interface (i.e., I/O interface) is also connected to the bus.

The following components are connected to the input/output interface: an input section including a keyboard, a mouse, and the like; an output section including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section including a hard disk and the like; and a communication section including a network interface card such as a local area network card, a modem, or the like. The communication section performs communication processing via a network such as the internet. The driver is also connected to the input/output interface as needed. A removable medium such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive as necessary, so that a computer program read out therefrom is mounted into the storage section as necessary.

In particular, according to embodiments of the present application, the processes described in the various method flowcharts may be implemented as computer software programs. For example, embodiments of the present application include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section, and/or installed from a removable medium. When executed by the central processing unit, performs various functions defined in the system of the present application.

According to an aspect of the present application, there is provided a computer-readable storage medium from which a processor of a computer device reads computer instructions, the processor executing the computer instructions to cause the computer device to perform the method provided in the above-mentioned various alternative implementations.

Alternatively, in the present embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the steps of:

and S4, under the condition that abnormal log data exist in the deep log data, calling the original log data matched with the abnormal log data from the second database to perform abnormal investigation processing.

Alternatively, in this embodiment, a person skilled in the art may understand that all or part of the steps in the methods of the foregoing embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, read-Only memories (ROMs), random Access Memories (RAMs), magnetic or optical disks, and the like.

The above-mentioned serial numbers of the embodiments of the present application are merely for description, and do not represent the advantages and disadvantages of the embodiments.

The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or a part of or all or part of the technical solution contributing to the prior art may be embodied in the form of a software product stored in a storage medium, and including instructions for causing one or more computer devices (which may be personal computers, servers, network devices, or the like) to execute all or part of the steps of the method of the embodiments of the present application.

In the embodiments of the present application, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to the related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other ways. The above-described apparatus embodiments are merely illustrative, and for example, a division of a unit is only a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed coupling or direct coupling or communication connection between each other may be an indirect coupling or communication connection through some interfaces, units or modules, and may be electrical or in other forms.

Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may also be implemented in the form of a software functional unit.

The foregoing is only a preferred embodiment of the present application and it should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present application, and these improvements and modifications should also be considered as the protection scope of the present application.

Claims

1. A log processing method, comprising:

and calling the original log data matched with the abnormal log data from the second database to perform abnormal investigation processing under the condition that the abnormal log data exists in the deep log data.

2. The method according to claim 1, wherein in a case where a system log generated by an application system is obtained, performing a statistical processing on the system log to obtain basic log data includes:

under the condition of acquiring the system log, extracting the content of the system log to obtain the content of the system log;

and counting the system log according to the system log content to obtain the basic log data.

3. The method of claim 2, wherein the performing statistics on the system log according to the system log content to obtain the basic log data comprises:

classifying the system log according to the basic attribute type of the system log content to obtain a classification result;

and counting the classification result to obtain basic log data under each basic attribute type.

4. The method according to claim 1, wherein in a case that abnormal log data exists in the deep log data, the invoking, from the second database, the original log data matched with the abnormal log data for exception checking processing comprises:

under the condition that the abnormal log data exist in the deep log data, acquiring a log data identifier corresponding to the abnormal log data;

indexing the original log data stored in the second database by using the log data identifier to obtain original log data matched with the log data identifier;

and calling the original log data matched with the log data identifier to perform exception troubleshooting.

5. The method of claim 4, wherein invoking the original log data with the log data identity matching for exception troubleshooting comprises:

acquiring a plurality of process logs recorded in original log data matched with the log data identification, wherein the process logs are logs generated when the application system runs to each process;

determining an abnormal flow log matched with the abnormal log data from the plurality of flow logs;

and determining the process corresponding to the abnormal process log in the application system as an abnormal process.

6. The method according to any one of claims 1 to 5, wherein before said performing a statistical processing on said system log to obtain basic log data, said method comprises at least one of the following;

acquiring the system log uploaded by the application system through a log gateway;

and acquiring the system log written into a log directory by the application system according to a preset rule.

7. The method of any one of claims 1 to 5, wherein after storing the base log data in a first database and storing original log data corresponding to the system log in a second database, the method comprises:

displaying a first identifier of the first database and a second identifier of the second database on a system interface;

responding to a first display instruction triggered by the first identifier, and displaying the basic log data; or, responding to a second display instruction triggered by the second identifier, and displaying the original log data.

8. A log processing apparatus, comprising:

the second processing unit is used for responding to a statistical analysis instruction, calling basic log data corresponding to the statistical analysis instruction from the first database to carry out secondary statistical processing, and obtaining deep log data;

and the third processing unit is used for calling the original log data matched with the abnormal log data from the second database to perform abnormal investigation processing under the condition that the abnormal log data exists in the deep log data.

9. A computer-readable storage medium, comprising a stored program, wherein the program when executed performs the method of any of claims 1 to 7.

10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 7 by means of the computer program.