CN113918526A

CN113918526A - Log processing method and device, computer equipment and storage medium

Info

Publication number: CN113918526A
Application number: CN202111187618.XA
Authority: CN
Inventors: 史银波
Original assignee: Ping An International Smart City Technology Co Ltd
Current assignee: Shenzhen Ping An Smart Healthcare Technology Co ltd
Priority date: 2021-10-12
Filing date: 2021-10-12
Publication date: 2022-01-11
Anticipated expiration: 2041-10-12
Also published as: CN113918526B

Abstract

The application relates to the technical field of artificial intelligence, and provides a log processing method, a device, computer equipment and a storage medium, wherein the method comprises the following steps: carrying out log collection on a service system to obtain a log; sending the log to a service log file; judging whether a log viewing request input by a user through a front-end page is received or not; if so, carrying out authority verification on the user based on the user information; if the authority passes the verification, acquiring a target log corresponding to the service type index from the service log file based on a preset shell script and the service type index; establishing a webSocket connecting channel between a log server and a front-end page; and displaying the target log on a front page through the webSocket connecting channel. By the method and the device, the logs can be pushed to the front-end page in real time, and the flexibility and intelligence of real-time log checking are improved. The method and the device can also be applied to the field of block chains, and the data such as the target logs can be stored on the block chains.

Description

Log processing method and device, computer equipment and storage medium

Technical Field

The application relates to the technical field of artificial intelligence, in particular to a log processing method and device, computer equipment and a storage medium.

Background

For better monitoring and analyzing logs, more log viewing platforms, such as ELK log viewing platform in local area network (logstack + elastic search + Kibana4), are present nowadays. The ELK log viewing platform is in a B/S form, and logs scattered in different places can be directly queried through a browser. However, the ELK log viewing platform has poor real-time performance when analyzing logs, and cannot perform real-time log viewing, which makes the ELK log viewing platform inconvenient to use in some scenes with strong real-time performance, lacks the capability of pushing logs to a user browser in real time, cannot well meet the requirement of timeliness required by real-time log viewing, and has low flexibility and intelligence for log viewing.

Disclosure of Invention

The application mainly aims to provide a log processing method, a log processing device, computer equipment and a storage medium, and aims to solve the technical problems that an existing ELK log viewing platform cannot well meet the timeliness requirement required by real-time log viewing, and the flexibility and intelligence of log viewing are low.

The application provides a log processing method, which comprises the following steps:

carrying out log collection on a preset service system through a first preset log collection component to obtain a corresponding log; wherein the log comprises service logs of a plurality of different business services;

sending the log to a service log file through a second preset log collection component; the service log file is a file in a preset log server, and service logs in the service log file are stored in an index mode;

judging whether a log viewing request input by a user through a front-end page is received or not; the log viewing request carries a service type index and user information;

if the log checking request is received, based on the user information, calling a preset role authority level data table and a preset business operation authority level data table to carry out authority verification on the user, and judging whether the authority verification passes;

if the authority passes the verification, acquiring a target log corresponding to the service type index from the service log file based on a preset shell script and the service type index;

establishing a webSocket connecting channel between the log server and the front-end page;

and displaying the target log on the front-end page through the webSocket connecting channel.

Optionally, the step of obtaining a target log corresponding to the service type index from the service log file based on a preset shell script and the service type index includes:

acquiring the shell script and acquiring the service type index;

filling the service type index to a corresponding position in the shell script to obtain the filled shell script;

executing the filled shell script, and searching all associated service indexes corresponding to the service type indexes from the service log file;

based on the associated service indexes, extracting designated logs corresponding to the associated service indexes from the service log file;

and taking all the designated logs as the target logs.

Optionally, the step of displaying the target log on the front-end page through the webSocket connection channel includes:

pushing a target log to a client corresponding to the front-end page through the webSocket connecting channel;

rendering the target log stored on the client according to a preset rendering rule to obtain a rendered target log;

and displaying the rendered target log on the front-end page according to a preset display rule.

Optionally, the step of calling a preset role permission level data table and a preset service operation permission level data table to perform permission verification on the user based on the user information, and determining whether the permission verification passes includes:

calling the role authority level data table and the service operation authority level data table;

judging whether designated user information which is the same as the user information is stored in the role authority level data table or not;

if the appointed user information is stored, inquiring a target authority level corresponding to the appointed user information based on the role authority level data table, and inquiring authority limit information corresponding to the appointed user information;

inquiring the service authority level of the service operation corresponding to the viewing log data based on the service operation authority level data table;

judging whether the target permission level is greater than the service permission level;

if the service authority level is higher than the service authority level, acquiring current time information;

judging whether the time information is in a time range corresponding to the authority deadline information;

and if the authority limit information is in the time range corresponding to the authority limit information, judging that the authority verification is passed, otherwise, judging that the authority verification is not passed.

Optionally, the target log includes operation data of the service system, and after the step of presenting the target log on the front-end page through the webSocket connection channel, the method includes:

acquiring running data corresponding to the service system from the target log; wherein the amount of operational data comprises a plurality of items;

integrating the operation data to obtain a corresponding operation parameter set;

inputting the operation parameter set into a pre-trained abnormal prediction model, and constructing a feature vector corresponding to the operation parameter set through a feature extraction layer in the abnormal prediction model;

performing classified prediction on the feature vectors through a classification layer in the abnormal prediction model to obtain an operation label corresponding to the preset service system;

and generating an operation state corresponding to the service system based on the operation label.

Optionally, before the step of inputting the operating parameter set into a pre-trained abnormal prediction model and constructing a feature vector corresponding to the operating parameter set through a feature extraction layer in the abnormal prediction model, the method includes:

acquiring sample data acquired in advance; the sample data comprises a preset number of sample operation parameter sets and sample operation labels for labeling the sample operation parameter sets;

training a preset neural network model through the sample operation parameter set and the sample operation label to obtain a corresponding initial anomaly prediction model;

acquiring preset test data, and carrying out prediction precision test on the initial anomaly prediction model based on the test data to obtain corresponding prediction accuracy;

judging whether the prediction accuracy is greater than a preset accuracy threshold;

and if so, taking the initial abnormal prediction model as the abnormal prediction model.

Optionally, after the step of exposing the target log on the front-end page through the webSocket connection channel, the method includes:

after the target log passes through a preset display time length on the front-end page, judging whether an operation triggered by the user on the front-end page is received within a preset time length;

if the operation on the front-end page is not received, acquiring a memory occupied by the data of the target log;

judging whether the data occupied memory value is smaller than a preset data occupied memory threshold value or not;

if the data occupation memory threshold is smaller than the data occupation memory threshold, moving the target log in the front-end page to a local preset database for storage;

and if the data occupation memory threshold is not less than the data occupation memory threshold, moving the target log in the front-end page to a block chain for storage.

The present application further provides a log processing apparatus, including:

the acquisition module is used for acquiring logs of a preset service system through the first preset log acquisition component to obtain corresponding logs; wherein the log comprises service logs of a plurality of different business services;

the sending module is used for sending the log to a service log file through a second preset log collecting component; the service log file is a file in a preset log server, and service logs in the service log file are stored in an index mode;

the first judgment module is used for judging whether a log checking request input by a user through a front-end page is received or not; the log viewing request carries a service type index and user information;

the verification module is used for calling a preset role authority level data table and a service operation authority level data table to carry out authority verification on the user based on the user information if the log viewing request is received, and judging whether the authority verification passes;

the first obtaining module is used for obtaining a target log corresponding to the service type index from the service log file based on a preset shell script and the service type index if the authority verification is passed;

the creating module is used for creating a webSocket connecting channel between the log server and the front-end page;

and the display module is used for displaying the target log on the front-end page through the webSocket connecting channel.

The present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

The present application also provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the above-mentioned method.

The log processing method, the log processing device, the computer equipment and the storage medium have the following beneficial effects:

according to the log processing method, the log processing device, the computer equipment and the storage medium, after the log acquisition component is used for acquiring logs of a service system to obtain logs, the logs are firstly sent into a service log file, after a log viewing request input by a user through a front-end page is received and the user is judged to pass permission verification, a target log corresponding to a service type index is obtained from the service log file based on a preset shell script and the service type index, then a webSocket connecting channel is established between a log server and the front-end page, and finally the target log is displayed on the front-end page through the webSocket connecting channel. According to the method and the device, the log viewing platform is provided with the capability of pushing the log to the front-end page in real time through a technical combination mode based on the log viewing platform, the webSocket and the shell script, and the flexibility and the intelligence of the log viewing platform about real-time log viewing are improved. In addition, the user can check the streaming log in real time only by inputting the corresponding service type index according to the self requirement, so that great flexibility of log checking is provided for the user, and the use experience of the user is improved.

Drawings

FIG. 1 is a flowchart illustrating a log processing method according to an embodiment of the present application;

FIG. 2 is a schematic structural diagram of a log processing apparatus according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of a computer device according to an embodiment of the present application.

The implementation, functional features and advantages of the objectives of the present application will be further explained with reference to the accompanying drawings.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.

The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Referring to fig. 1, a log processing method according to an embodiment of the present application includes:

s10: carrying out log collection on a preset service system through a first preset log collection component to obtain a corresponding log; wherein the log comprises service logs of a plurality of different business services;

s20: sending the log to a service log file through a second preset log collection component; the service log file is a file in a preset log server, and service logs in the service log file are stored in an index mode;

s30: judging whether a log viewing request input by a user through a front-end page is received or not; the log viewing request carries a service type index and user information;

s40: if the log checking request is received, based on the user information, calling a preset role authority level data table and a preset business operation authority level data table to carry out authority verification on the user, and judging whether the authority verification passes;

s50: if the authority passes the verification, acquiring a target log corresponding to the service type index from the service log file based on a preset shell script and the service type index;

s60: establishing a webSocket connecting channel between the log server and the front-end page;

s70: and displaying the target log on the front-end page through the webSocket connecting channel.

As described in the above steps S10 to S70, the main execution body of the embodiment of the method is a log processing device. In practical applications, the log processing apparatus may be implemented by a virtual apparatus, such as a software code, or by an entity apparatus written or integrated with a relevant execution code, and may perform human-computer interaction with a user through a keyboard, a mouse, a remote controller, a touch panel, or a voice control device. The log processing device in the embodiment can push the log to the front-end page in real time, and improves the flexibility and intelligence of real-time log checking. Specifically, firstly, a first preset log collection component is used for collecting logs of a preset service system to obtain a corresponding log. Wherein the log comprises service logs of a plurality of different business services. In addition, the first preset log collection component may be a filebed log collection tool. The business system can be a system determined according to actual business requirements, and can be a medical business system and the like. The business services may include user services, order services, and the like.

And then the log is sent to a service log file through a second preset log collection component. The service log file is a file in a preset log server, and service logs in the service log file are stored in an index mode. In addition, the second preset log collection component may specifically be a logstack log collection server. The FileBeat log collection tool can be used for collecting all initial logs in the service system, then the collected initial logs are sent to a LogStash log collection server for filtering processing to obtain processed logs, and the LogStash log collection server sends the processed logs to a service log file in a log server. The log server may specifically be an Elasticsearch engine server, and the service log file is a fixed log file for storing a service log. The name of the service may be used as an index of the corresponding service log (also referred to as a service index), and the service log may be stored in the log server based on the service index of the service log. In addition, the fileBeat log collection tool, the LogStash log collection server and the Elasticissearch search engine server in the application can be collectively referred to as an ELK log viewing platform. In addition, after the service indexes are set for the service logs in the log server, index rules are further configured in the log server in advance according to actual aggregated service requirements, and log output rules corresponding to the index rules are configured at the same time. The index rule specifies the corresponding relationship between the service index of the next level of the same class and the service type index, that is, one service type index has the corresponding relationship with all the associated service indexes corresponding to the service type index, and all the service indexes of the same class are classified into the same service type index. For example, for the service indexes of user service 1, user service 2, and user service 3, the service indexes can be categorized into one same service type index "user service". The log output rule refers to that for a received service type index, logs of all associated service indexes contained in the service type index are correspondingly output. After the log server completes the configuration of the index rule, the log server can synchronize the index rule to a LogStash search engine server. And then the LogStash search engine server sends the log, the index rule and the log output rule to a service log file in a preset log server. When the service type index input by the user is received, the device reads the index rule and the log output rule stored in the log server, and further queries a target log corresponding to the service type index from the service log file. For example, if the service type index is the user service, and the service log file includes the log with the service index as the user service 1 and the log with the service index as the user service 2, the log of the user service 1 and the log of the user service 2 are found from the service log file as the target logs based on the index rule and the log output rule. By carrying out aggregation query on the service type indexes in the log server, the source distribution condition of the log is not considered, and thus the streaming viewing of any service under multi-instance deployment can be realized simply and quickly.

And then judging whether a log viewing request input by a user through the front-end page is received. The log viewing request carries a service type index and user information. The front-end page may be a browser in a client. The log viewing request refers to a request for inquiring a service log which needs to be viewed by a user. The service type index refers to index information of a service log required to be viewed. The user information may include user id or user name information. And if the log checking request is received, calling a preset role authority level data table and a preset business operation authority level data table to carry out authority verification on the user based on the user information, and judging whether the authority verification passes. For the specific implementation process of the authority verification of the user by invoking the preset role authority level data table and the service operation authority level data table, this will be further described in the subsequent specific embodiments, and details are not repeated here

And if the authority passes the verification, acquiring a target log corresponding to the service type index from the service log file based on a preset shell script and the service type index. The service type index can be filled to a corresponding position in the shell script to obtain the filled shell script; executing the filled shell script, and searching all associated service indexes corresponding to the service type indexes from the service log file; and further extracting specified logs corresponding to the associated service indexes from the service log file based on the associated service indexes, and taking all the specified logs as the target logs.

And subsequently establishing a webSocket connecting channel between the log server and the front-end page. The webSocket is a Protocol for performing full duplex communication on a single TCP (Transmission Control Protocol) connection. The webSocket allows the back-end server to actively push data to the client. In the webSocket protocol, a client browser (such as a front-end page) and a back-end server only need to complete one handshake to establish a persistent connection, and bidirectional data transmission is performed between the client browser and the back-end server. The webSocket protocol is different from the HTTP protocol in the most obvious characteristic that the webSocket protocol can be initiatively initiated by a server side, and is very suitable for a scene that a browser needs to receive data changes in time. And finally, displaying the target log on the front-end page through the webSocket connecting channel. The display mode of the target log is not limited, and can be set according to actual requirements.

In this embodiment, after the log collection component is used to collect logs of the service system to obtain logs, the logs are sent to a service log file, after a log viewing request input by a user through a front-end page is received and the user is judged to pass through authority verification, a target log corresponding to a service type index is obtained from the service log file based on a preset shell script and the service type index, a webSocket connection channel is established between a log server and the front-end page, and finally the target log is displayed on the front-end page through the webSocket connection channel. According to the embodiment, through a technical combination mode based on the log viewing platform (including the log collecting assembly and the log server) + webSocket + shell script, the capability of pushing the log to a front-end page in real time is provided for the log viewing platform, and the flexibility and the intelligence of the log viewing platform about real-time log viewing are improved. In addition, the user can check the streaming log in real time only by inputting the corresponding service type index according to the self requirement, so that great flexibility of log checking is provided for the user, and the use experience of the user is improved.

Further, in an embodiment of the present application, the step S50 includes:

s500: acquiring the shell script and acquiring the service type index;

s501: filling the service type index to a corresponding position in the shell script to obtain the filled shell script;

s502: executing the filled shell script, and searching all associated service indexes corresponding to the service type indexes from the service log file;

s503: based on the associated service indexes, extracting designated logs corresponding to the associated service indexes from the service log file;

s504: and taking all the designated logs as the target logs.

As described in the foregoing steps S500 to S504, the step of obtaining, based on the preset shell script and the service type index, the target log corresponding to the service type index from the service log file may specifically include: firstly, the shell script is obtained, and the service type index is obtained. The shell script is a pre-written and generated script including a tail command, and specific content of the shell script can be set according to actual requirements, for example, the shell script may be: log, xxx is an index of service types that need to be populated. And then filling the service type index to a corresponding position in the shell script to obtain the filled shell script. The corresponding position in the shell script is a position corresponding to the service type index that needs to be filled, such as xxx. And then executing the filled shell script, and searching all associated service indexes corresponding to the service type indexes from the service log file. For example, if the service type index is a user service, and the service log file includes an index 1: user service 1, index 2: and if the user service 2 is the user service, the index 1 and the index 2 are associated service indexes corresponding to the service type indexes of the user service. And subsequently, based on the associated service indexes, extracting the designated logs respectively corresponding to the associated service indexes from the service log file. The service log file comprises service type indexes and logs stored in the service log file, wherein the logs stored in the service log file have one-to-one corresponding storage relation with the service type indexes, and further specified logs corresponding to the associated service indexes can be extracted from the service log file according to the storage relation. And finally, taking all the designated logs as the target logs. In this embodiment, after the service type index is obtained, a target log corresponding to the service type index can be obtained from the service log file in real time by executing a preset shell script, so that the target log can be quickly and conveniently obtained. In addition, as only one service type index needs to be input by the user, the appointed logs related to all associated service indexes corresponding to the service type indexes can be intelligently searched from the log server, and all the obtained appointed logs are aggregated to be output as the required target logs, so that the user does not need to input each associated service index, the input workload of the user is reduced, and the intelligence and the convenience of log acquisition are improved. In addition, when the target log is obtained, the target log can be displayed on the front-end page through the webSocket link channel subsequently, so that the target log subjected to service aggregation can be pushed to the front-end page such as a browser in real time, a user can check the target log in real time, great flexibility of log checking is provided for the user, and the use experience of the user is improved.

Further, in an embodiment of the present application, the step S70 includes:

s700: pushing a target log to a client corresponding to the front-end page through the webSocket connecting channel;

s701: rendering the target log stored on the client according to a preset rendering rule to obtain a rendered target log;

s702: and displaying the rendered target log on the front-end page according to a preset display rule.

As described in the foregoing steps S700 to S702, the step of displaying the target log on the front-end page through the webSocket connection channel may specifically include: firstly, a target log is pushed to a client corresponding to the front-end page through the webSocket connecting channel. And then rendering the target log stored on the client according to a preset rendering rule to obtain the rendered target log. Wherein, rendering the target log according to a preset rendering rule may refer to: and modifying the image-text format and the color of the target log to enhance the display effect, such as highlighting the target log. And finally, displaying the rendered target log on the front-end page according to a preset display rule. The process of displaying the rendered target log on the front-end page according to the preset display rule may include: and displaying the target log at a specified position of the front-end page according to a preset display rule, wherein the specified position includes but is not limited to an upper left area or an upper right area of the front-end page, and generally does not include a middle area of the front-end page, so as to avoid affecting the operation of a user on the client. In the embodiment, after the target log corresponding to the service type index is obtained, the target log is pushed to the client corresponding to the front-end page through the webSocket link channel, the target log stored on the client is rendered according to the preset rendering rule, and the rendered target log is displayed on the front-end page according to the preset display rule, so that the target log is pushed to the front-end page of a user such as a browser in real time, the user can view the streaming log in real time, great flexibility of log viewing is provided for the user, and the use experience of the user is improved.

Further, in an embodiment of the present application, the step S40 includes:

s400: calling the role authority level data table and the service operation authority level data table;

s401: judging whether designated user information which is the same as the user information is stored in the role authority level data table or not;

s402: if the appointed user information is stored, inquiring a target authority level corresponding to the appointed user information based on the role authority level data table, and inquiring authority limit information corresponding to the appointed user information;

s403: inquiring the service authority level of the service operation corresponding to the viewing log data based on the service operation authority level data table;

s404: judging whether the target permission level is greater than the service permission level;

s405: if the service authority level is higher than the service authority level, acquiring current time information;

s406: judging whether the time information is in a time range corresponding to the authority deadline information;

s407: and if the authority limit information is in the time range corresponding to the authority limit information, judging that the authority verification is passed, otherwise, judging that the authority verification is not passed.

As described in the foregoing steps S400 to S407, the step of invoking a preset role permission level data table and a preset service operation permission level data table to perform permission verification on the user based on the user information, and determining whether the permission verification passes may specifically include: firstly, calling the role authority level data table and the service operation authority level data table. The role authority level data table is a first data table which is created in advance and records legal user information, authority levels corresponding to the legal user information one by one and authority limit information corresponding to the authority levels of all the employees one by one. The service operation permission level data table is a second data table which is created in advance and records various service operations and service permission levels corresponding to the various service operations one by one. And then judging whether the designated user information which is the same as the user information is stored in the role authority level data table. And if the appointed user information is stored, inquiring a target authority level corresponding to the appointed user information based on the role authority level data table, and inquiring authority limit information corresponding to the appointed user information. The authority limit information is information corresponding to temporal limitation of the authority level of the user. For example, it may be specified that a user of a certain identity has certain rights for a certain period of time, or that a user of a certain identity uses a certain right for a certain length of time. And then inquiring the service authority level of the service operation corresponding to the viewing log data based on the service operation authority level data table. And subsequently judging whether the target permission level is greater than the service permission level. And if the service authority level is higher than the service authority level, acquiring the current time information. And finally, judging whether the time information is in a time range corresponding to the authority deadline information. Wherein the current time information is verified in order to determine the validity period of the target permission level of the target user. And if the current time information is in the time range corresponding to the authority limit information, indicating that the target authority level of the target user is still in a valid state currently. And if the current time information is not in the time range corresponding to the authority limit information, the target authority level of the target user is in a failure state currently. And if the authority limit information is in the time range corresponding to the authority limit information, judging that the authority verification is passed, otherwise, judging that the authority verification is not passed. In the embodiment, the target authority level and the authority limit information of the user are obtained by inquiring the role authority level data table, the service authority level of the service operation corresponding to the log data to be checked is inquired through the service operation authority level data table, then the target authority level and the service authority level are compared, and whether the time information is in the verification processing in the time range corresponding to the authority limit information is judged, so that whether the user has the authority to check the log data can be intelligently and quickly judged according to the obtained processing result, the condition that the service for checking the log data is provided for illegal users or users without authority can be effectively avoided, the safety of the log data is effectively ensured, and the processing safety of log checking is improved.

Further, in an embodiment of the present application, the target log includes operation data of the service system, and after the step S70, the method includes:

s710: acquiring running data corresponding to the service system from the target log; wherein the amount of operational data comprises a plurality of items;

s711: integrating the operation data to obtain a corresponding operation parameter set;

s712: inputting the operation parameter set into a pre-trained abnormal prediction model, and constructing a feature vector corresponding to the operation parameter set through a feature extraction layer in the abnormal prediction model;

s713: performing classified prediction on the feature vectors through a classification layer in the abnormal prediction model to obtain an operation label corresponding to the preset service system;

s714: and generating an operation state corresponding to the service system based on the operation label.

As described in the foregoing steps S710 to S714, the target log includes the operation data of the service system, and after the step of presenting the target log on the front-end page through the webSocket connection channel is completed, a process of detecting the operation state of the service system based on the operation data of the service system included in the target log may also be included. Specifically, first, the operation data corresponding to the service system is acquired from the target log. Wherein the amount of the operational data comprises a plurality of items. The operation data is data extracted from a log generated by the business system and used to reflect the operation state of the system, and includes information such as user login time, user logged in, request type, processing result (success or failure) of the system, response time of the system to a request, and cause of failure in processing a request by the system. A preset operation data identifier may be obtained first, and then data corresponding to the operation data identifier may be obtained from a target log as the operation data. Specifically, the service system is provided with a plurality of task nodes, so that in the operation process of the service system, corresponding logical operations, such as receiving a request, processing the request, issuing an instruction, transmitting data, and the like, are executed at each task node. In the operation process of the service system, logs are correspondingly generated according to the performed logical operations, for example, a request receiving log generated when a request is received, a user login log generated when a user logs in, a processing success log for successfully processing the request, and the like. The logs generated by the system may thus be directly related to the operation of the system. In the operation process of the service system, the operation condition of the system can be known through information such as system request processing failure information, database read-write errors, service request quantity processed by the system and the like. And then integrating the operation data to obtain a corresponding operation parameter set. In order to accurately reflect the operation state of the service system, the operation data of a single dimension cannot accurately reflect, so that an operation parameter set needs to be obtained by analyzing and counting a plurality of operation data, and the operation state of the service system is reflected from the operation data of a plurality of dimensions. And then inputting the operation parameter set into a pre-trained abnormal prediction model, and constructing a feature vector corresponding to the operation parameter set through a feature extraction layer in the abnormal prediction model. The abnormal prediction model is generated by training a neural network model based on sample data acquired in advance according to actual use requirements, and the adopted neural network model can be any one of a cyclic neural network model, a random forest network model, a convolutional neural network model and a recurrent neural network model, which is not specifically limited herein. And after the neural network model is trained through the training samples, obtaining an abnormal prediction model, namely identifying the operation state of the specific operation parameter set related to the service system through the abnormal prediction model. The operation state identification of the service system is used for identifying whether the service system operates normally. In addition, the operation parameter set comprises a plurality of operation data of the service system, and the operation data are associated with each other, so that the operation state identification is carried out according to the operation parameter set through the abnormal prediction model, and the operation state of the system is determined without analyzing each operation data in the operation parameter set, thereby realizing the efficient and rapid identification of the operation state of the service system. In addition, the characteristic vector is constructed according to each operating data in the operating parameter set and is used for reflecting the characteristics of the operating parameter set. In order to determine the operation state of the business system, an operation label corresponding to each operation state is configured in the anomaly prediction model. For example, if the operation state of the service system includes a normal operation state and an abnormal operation state, the operation label indicating the normal operation of the service system is configured for the normal operation state of the service system, and the operation label indicating the abnormal operation of the service system is configured for the abnormal operation filling of the service system. For the operation states of the service system including more than two operation states, an operation label is correspondingly configured for each operation state, and the operation labels are in one-to-one correspondence with the operation states, namely, one operation label uniquely corresponds to one operation state, and one operation state also uniquely corresponds to one operation label. And subsequently, carrying out classification prediction on the feature vectors through a classification layer in the abnormal prediction model to obtain an operation label corresponding to the preset service system. The feature vectors are classified and predicted, namely the probability of the feature vectors corresponding to each operation label is obtained through calculation and analysis, then the probability of traversing each operation label is carried out, and the operation label with the maximum probability is used as the operation label predicted according to the operation parameter set. In addition, the operation labels and the operation states are in one-to-one correspondence, so that the operation state of the service system can be determined according to the predicted operation labels. And finally, generating an operation state corresponding to the service system based on the operation label. Further, if the operation state indicates that the service system is abnormally operated, early warning prompt for the service system can be performed. The early warning prompt can be the generation of early warning prompt information and the transmission of the early warning prompt information to the current page, so that the user is timely reminded of the abnormal operation condition of the current business system, the user can take measures in time, more serious problems are avoided, and the abnormal condition can be timely found and solved. In the embodiment, after the target log corresponding to the service system is obtained, the operation parameter set is obtained through statistics according to a plurality of items of operation data extracted from the target log, and the operation state of the service system is identified according to the operation parameter set through the anomaly prediction model.

Further, in an embodiment of the present application, before the step S712, the method includes:

s7120: acquiring sample data acquired in advance; the sample data comprises a preset number of sample operation parameter sets and sample operation labels for labeling the sample operation parameter sets;

s7121: training a preset neural network model through the sample operation parameter set and the sample operation label to obtain a corresponding initial anomaly prediction model;

s7122: acquiring preset test data, and carrying out prediction precision test on the initial anomaly prediction model based on the test data to obtain corresponding prediction accuracy;

s7123: judging whether the prediction accuracy is greater than a preset accuracy threshold;

s7124: and if so, taking the initial abnormal prediction model as the abnormal prediction model.

As described in steps S7120 to S7124, before the step of inputting the operation parameter set into a pre-trained abnormal prediction model and constructing a feature vector corresponding to the operation parameter set through a feature extraction layer in the abnormal prediction model is performed, a process of training and generating the abnormal prediction model may be further included. Specifically, first, sample data acquired in advance is acquired. The sample data comprises a preset number of sample operation parameter sets and sample operation labels for labeling the sample operation parameter sets. And then training a preset neural network model through the sample operation parameter set and the sample operation label to obtain a corresponding initial anomaly prediction model. In the training process, the neural network model carries out classification prediction on each sample operation parameter set to obtain an operation label of the sample operation parameter set, if the predicted operation label is different from the labeled sample operation label, the parameters of the neural network model are adjusted until the predicted operation label is consistent with the labeled sample operation label, and a corresponding trained initial anomaly prediction model is obtained. And then, acquiring preset test data, and carrying out prediction precision test on the initial anomaly prediction model based on the test data to obtain a corresponding prediction accuracy. The data of a preset proportion can be provided from the sample data as the test data, and the preset proportion is not limited and can be determined according to actual requirements, for example, the preset proportion can be 30%. And finally, judging whether the prediction accuracy is greater than a preset accuracy threshold. And if the initial abnormal prediction model is larger than the accuracy threshold, taking the initial abnormal prediction model as the abnormal prediction model. Specifically, after training a neural network model for a period of time to obtain an initial anomaly prediction model, performing prediction precision test on the initial anomaly prediction model by using the test data, namely inputting a plurality of test sample operation parameter sets into the initial anomaly prediction model, predicting by the initial anomaly prediction model to obtain an operation label of each test sample operation parameter set, comparing the operation label of each test sample operation parameter set with a sample operation label labeled for the test sample operation parameter set, if the operation labels are consistent, accurately predicting the test sample operation parameter set by the initial anomaly prediction model, and if the operation labels are inconsistent, predicting by the initial anomaly prediction model to the test sample operation parameter set, thereby obtaining the prediction accuracy of the initial anomaly prediction model by statistics (the prediction accuracy is the number of the test sample operation parameter sets with accurate prediction/the total number of the test sample operation parameter sets Quantity), if the obtained prediction accuracy meets the set accuracy requirement, namely is greater than a preset accuracy threshold, the initial abnormal prediction model is converged, so that the training of the initial abnormal prediction model is finished, and the trained initial abnormal prediction model is used as an abnormal prediction model and used for identifying the operation state of the operation parameter set. In addition, the neural network model is trained through a preset number of sample operation parameter sets and corresponding sample operation labels, so that the operation state identification precision of the neural network model can be guaranteed. Further, if the prediction accuracy is less than or equal to the accuracy threshold, it indicates that the training of the trained initial abnormal prediction model has not reached the preset standard, and may be that the number of samples of sample data used for training is too small or the number of samples of test data is too small, so in this case, the preset number is increased (i.e. the number of samples is increased, for example, a fixed number is increased each time or a random number is increased each time), then the training steps and the testing steps are executed again on the basis, and the model training is ended in this loop until the requirement that the model accuracy of the trained initial abnormal prediction model is greater than the preset accuracy threshold is reached. In the embodiment, the preset neural network model is trained and verified based on the sample data comprising the preset number of sample operation parameter sets and the sample operation labels marked for the sample operation parameter sets, so that an abnormal prediction model meeting the actual use requirement can be intelligently and quickly generated, the abnormal analysis of the operation data based on the abnormal prediction model can be favorably realized when the operation data corresponding to the service system is obtained subsequently, the operation state corresponding to the service system can be accurately and quickly generated, and corresponding measures can be taken for the service system according to the operation state so as to ensure the safe operation of the service system.

Further, in an embodiment of the present application, after the step S70, the method includes:

s720: after the target log passes through a preset display time length on the front-end page, judging whether an operation triggered by the user on the front-end page is received within a preset time length;

s721: if the operation on the front-end page is not received, acquiring a memory occupied by the data of the target log;

s722: judging whether the data occupied memory value is smaller than a preset data occupied memory threshold value or not;

s723: if the data occupation memory threshold is smaller than the data occupation memory threshold, moving the target log in the front-end page to a local preset database for storage;

s724: and if the data occupation memory threshold is not less than the data occupation memory threshold, moving the target log in the front-end page to a block chain for storage.

As described in the above steps S720 to S724, after the step of exposing the target log on the front-end page through the webSocket connection channel is completed, a mobile storage process for the target log on the front-end page may be further included. Specifically, after the target log passes through a preset display duration on the front-end page, it is first determined whether an operation triggered by the user on the front-end page is received within a preset duration. The specific value of the preset display duration is not limited, and can be set according to actual requirements, for example, the setting time can be 3 minutes. Similarly, the specific value of the preset duration is not limited, and may be set according to actual requirements, for example, may be set to 5 minutes. In addition, if the operation of the front-end page triggered by the user is not received within the preset time length, it can be judged that the user does not have the requirement for continuously viewing the target log at present. And if the operation on the front-end page is not received, acquiring a memory occupied data value of the target log. And then judging whether the data occupied memory value is smaller than a preset data occupied memory threshold value. And if the data occupation rate is less than the memory occupation threshold value, moving the target log in the front-end page to a local preset database for storage. And if the data occupation memory threshold is not less than the data occupation memory threshold, moving the target log in the front-end page to a block chain for storage. The specific value of the data occupied memory value can be set according to actual requirements, if the data occupied memory value of the target log is larger than the data occupied memory threshold value, the target log is considered to be stored locally, so that the normal operation of the device is affected, and the target log is further stored on the block chain, and the storage intelligence and the storage safety of the target log are improved. In the embodiment, when it is judged that the user does not have the requirement for viewing the target log at present, the target log can be stored in the local database or the block chain correspondingly according to the memory occupied by the data of the target log, so that the target log cannot occupy the position of a front-end page, a display position can be reserved for the following log pushed from the log server, the target program log is stored in the corresponding storage position, management and acquisition are facilitated, and the processing intelligence of the target log is effectively improved.

The log processing method in the embodiment of the present application may also be applied to the field of block chains, for example, data such as the target log is stored on a block chain. By storing and managing the target log by using the block chain, the security and the non-tamper property of the target log can be effectively ensured.

The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism and an encryption algorithm. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.

The block chain underlying platform can comprise processing modules such as user management, basic service, intelligent contract and operation monitoring. The user management module is responsible for identity information management of all blockchain participants, and comprises public and private key generation maintenance (account management), key management, user real identity and blockchain address corresponding relation maintenance (authority management) and the like, and under the authorization condition, the user management module supervises and audits the transaction condition of certain real identities and provides rule configuration (wind control audit) of risk control; the basic service module is deployed on all block chain node equipment and used for verifying the validity of the service request, recording the service request to storage after consensus on the valid request is completed, for a new service request, the basic service firstly performs interface adaptation analysis and authentication processing (interface adaptation), then encrypts service information (consensus management) through a consensus algorithm, transmits the service information to a shared account (network communication) completely and consistently after encryption, and performs recording and storage; the intelligent contract module is responsible for registering and issuing contracts, triggering the contracts and executing the contracts, developers can define contract logics through a certain programming language, issue the contract logics to a block chain (contract registration), call keys or other event triggering and executing according to the logics of contract clauses, complete the contract logics and simultaneously provide the function of upgrading and canceling the contracts; the operation monitoring module is mainly responsible for deployment, configuration modification, contract setting, cloud adaptation in the product release process and visual output of real-time states in product operation, such as: alarm, monitoring network conditions, monitoring node equipment health status, and the like.

Referring to fig. 2, an embodiment of the present application further provides a log processing apparatus, including:

the acquisition module 1 is used for acquiring logs of a preset service system through a first preset log acquisition component to obtain corresponding logs; wherein the log comprises service logs of a plurality of different business services;

the sending module 2 is used for sending the log to a service log file through a second preset log collecting component; the service log file is a file in a preset log server, and service logs in the service log file are stored in an index mode;

the first judging module 3 is used for judging whether a log checking request input by a user through a front-end page is received or not; the log viewing request carries a service type index and user information;

the verification module 4 is used for calling a preset role authority level data table and a preset service operation authority level data table to perform authority verification on the user based on the user information if the log viewing request is received, and judging whether the authority verification passes;

the first obtaining module 5 is configured to obtain, based on a preset shell script and the service type index, a target log corresponding to the service type index from the service log file if the permission verification passes;

the creating module 6 is used for creating a webSocket connecting channel between the log server and the front-end page;

and the display module 7 is used for displaying the target log on the front-end page through the webSocket connecting channel.

In this embodiment, the operations executed by the modules or units respectively correspond to the steps of the log processing method in the foregoing embodiment one to one, and are not described herein again.

Further, in an embodiment of the present application, the first obtaining module 5 includes:

the first obtaining unit is used for obtaining the shell script and obtaining the service type index;

the filling unit is used for filling the service type index to a corresponding position in the shell script to obtain the filled shell script;

the execution unit is used for executing the filled shell script and searching all associated service indexes corresponding to the service type indexes from the service log file;

an extracting unit, configured to extract, from the service log file, a designated log corresponding to each of the associated service indexes based on the associated service index;

and the determining unit is used for taking all the designated logs as the target logs.

Further, in an embodiment of the present application, the display module 7 includes:

the pushing unit is used for pushing the target log to the client corresponding to the front-end page through the webSocket connecting channel;

the rendering unit is used for rendering the target log stored on the client according to a preset rendering rule to obtain a rendered target log;

and the display unit is used for displaying the rendered target log on the front-end page according to a preset display rule.

Further, in an embodiment of the present application, the verification module 4 includes:

the calling unit is used for calling the role authority level data table and the service operation authority level data table;

a first judging unit, configured to judge whether designated user information that is the same as the user information is stored in the role authority level data table;

the first query unit is used for querying a target permission level corresponding to the appointed user information and querying permission term information corresponding to the appointed user information based on the role permission level data table if the appointed user information is stored;

the second query unit is used for querying the service authority level of the service operation corresponding to the viewing log data based on the service operation authority level data table;

the second judging unit is used for judging whether the target authority level is greater than the service authority level;

the second obtaining unit is used for obtaining the current time information if the service authority level is higher than the service authority level;

a third judging unit, configured to judge whether the time information is within a time range corresponding to the authority deadline information;

and the judging unit is used for judging that the authority verification passes if the authority is in the time range corresponding to the authority deadline information, and otherwise, judging that the authority verification fails.

Further, in an embodiment of the present application, the target log includes operation data of the service system, and the log processing apparatus includes:

the second acquisition module is used for acquiring the running data corresponding to the service system from the target log; wherein the amount of operational data comprises a plurality of items;

the integration module is used for integrating the operation data to obtain a corresponding operation parameter set;

the construction module is used for inputting the operation parameter set into a pre-trained abnormal prediction model and constructing a feature vector corresponding to the operation parameter set through a feature extraction layer in the abnormal prediction model;

the prediction module is used for carrying out classification prediction on the feature vectors through a classification layer in the abnormal prediction model to obtain an operation label corresponding to the preset service system;

and the generating module is used for generating the running state corresponding to the business system based on the running label.

Further, in an embodiment of the present application, the log processing apparatus includes:

the third acquisition module is used for acquiring pre-acquired sample data; the sample data comprises a preset number of sample operation parameter sets and sample operation labels for labeling the sample operation parameter sets;

the training module is used for training a preset neural network model through the sample operation parameter set and the sample operation label to obtain a corresponding initial anomaly prediction model;

the test module is used for acquiring preset test data and testing the prediction accuracy of the initial abnormity prediction model based on the test data to obtain corresponding prediction accuracy;

the second judgment module is used for judging whether the prediction accuracy is greater than a preset accuracy threshold;

the third judging module is used for judging whether the operation triggered by the user on the front-end page is received within the preset time length after the target log passes through the preset display time length on the front-end page;

a fourth obtaining module, configured to obtain a memory occupied by data of the target log if an operation on the front-end page is not received;

the fourth judging module is used for judging whether the data occupied memory value is smaller than a preset data occupied memory threshold value or not;

the first storage module is used for moving the target log in the front-end page to a local preset database for storage if the target log is smaller than the data occupation memory threshold;

and the second storage module is used for moving the target log in the front-end page to a block chain for storage if the target log is not smaller than the memory occupied by the data.

Referring to fig. 3, a computer device, which may be a server and whose internal structure may be as shown in fig. 3, is also provided in the embodiment of the present application. The computer device comprises a processor, a memory, a network interface, a display screen, an input device and a database which are connected through a system bus. Wherein the processor of the computer device is designed to provide computing and control capabilities. The memory of the computer device comprises a storage medium and an internal memory. The storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and computer programs in the storage medium to run. The database of the computer device is used for storing logs, service log files, service type indexes, user information, shell scripts and target logs. The network interface of the computer device is used for communicating with an external terminal through a network connection. The display screen of the computer equipment is an indispensable image-text output equipment in the computer, and is used for converting digital signals into optical signals so that characters and figures are displayed on the screen of the display screen. The input device of the computer equipment is the main device for information exchange between the computer and the user or other equipment, and is used for transmitting data, instructions, some mark information and the like to the computer. The computer program is executed by a processor to implement a log processing method.

The processor executes the log processing method and comprises the following steps:

Those skilled in the art will appreciate that the structure shown in fig. 3 is only a block diagram of a part of the structure related to the present application, and does not constitute a limitation to the apparatus and the computer device to which the present application is applied.

An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a log processing method, and specifically:

To sum up, after the log collection component is used to collect logs of the service system to obtain logs, the logs are sent to a service log file in a log server, after a log viewing request input by a user through a front-end page is received and the user is judged to pass the authority verification, a target log corresponding to the service type index is obtained from the service log file based on a preset shell script and the service type index, a webSocket connection channel is established between the log server and the front-end page, and finally the target log is displayed on the front-end page through the webSocket connection channel. According to the embodiment of the application, the log viewing platform is provided with the capability of pushing the log to the front-end page in real time through a technical combination mode based on the log viewing platform, the webSocket and the shell script, and the flexibility and the intelligence of the log viewing platform about real-time log viewing are improved. In addition, the user can check the streaming log in real time only by inputting the corresponding service type index according to the self requirement, so that great flexibility of log checking is provided for the user, and the use experience of the user is improved.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. Any reference to memory, storage, database, or other medium provided herein and used in the examples may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double-rate SDRAM (SSRSDRAM), Enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.

The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application, or which are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.

Claims

1. A log processing method, comprising:

2. The log processing method according to claim 1, wherein the step of obtaining a target log corresponding to the service type index from the service log file based on a preset shell script and the service type index comprises:

acquiring the shell script and acquiring the service type index;

and taking all the designated logs as the target logs.

3. The log processing method according to claim 1, wherein the step of exposing the target log on the front-end page through the webSocket connection channel includes:

4. The log processing method according to claim 1, wherein the step of invoking a preset role permission level data table and a preset service operation permission level data table to perform permission verification on the user based on the user information and judging whether the permission verification passes comprises:

5. The log processing method according to claim 1, wherein the target log includes operation data of the service system, and after the step of exposing the target log on the front-end page through the webSocket connection channel, the method includes:

6. The log processing method according to claim 5, wherein before the step of inputting the operation parameter set into a pre-trained abnormal prediction model and constructing a feature vector corresponding to the operation parameter set by a feature extraction layer in the abnormal prediction model, the method comprises:

7. The log processing method according to claim 1, wherein after the step of exposing the target log on the front-end page through the webSocket connection channel, the method comprises:

8. A log processing apparatus, comprising:

9. A computer device comprising a memory and a processor, the memory having stored therein a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.