CN116361127A - Log data processing method and device and electronic equipment - Google Patents
Log data processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN116361127A CN116361127A CN202211682873.6A CN202211682873A CN116361127A CN 116361127 A CN116361127 A CN 116361127A CN 202211682873 A CN202211682873 A CN 202211682873A CN 116361127 A CN116361127 A CN 116361127A
- Authority
- CN
- China
- Prior art keywords
- log
- log data
- user
- data
- index
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title abstract description 6
- 238000000034 method Methods 0.000 claims abstract description 41
- 230000006399 behavior Effects 0.000 claims abstract description 33
- 230000000694 effects Effects 0.000 claims abstract description 24
- 238000013145 classification model Methods 0.000 claims abstract description 13
- 230000006870 function Effects 0.000 claims description 39
- 238000012545 processing Methods 0.000 claims description 39
- 230000008569 process Effects 0.000 claims description 16
- 238000011156 evaluation Methods 0.000 claims description 13
- 230000008520 organization Effects 0.000 claims description 10
- 230000002159 abnormal effect Effects 0.000 claims description 6
- 238000013475 authorization Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 abstract description 9
- 230000010354 integration Effects 0.000 abstract description 2
- 238000007726 management method Methods 0.000 description 7
- 238000010276 construction Methods 0.000 description 5
- 230000004927 fusion Effects 0.000 description 4
- 230000009897 systematic effect Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000001960 triggered effect Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 235000014510 cooky Nutrition 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012827 research and development Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3495—Performance evaluation by tracing or monitoring for systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention discloses a log data processing method and device and electronic equipment. The method comprises the following steps: classifying the log data of the target system based on the log general classification model to obtain system running log data and user operation behavior log data of the target system; classifying the system operation log data based on the log main body data to obtain a log main body classification result of the target system; and analyzing the target system according to the log main body classification result of the target system and the log data of the user operation behaviors to determine the running state of the target system. The invention solves the technical problem that the log data of the new and old business systems cannot be effectively classified and managed in the related technology, achieves the technical effects of more effectively classifying and managing the log data of the new and old business systems, effectively finding a silent system and meeting the log management integration requirement of the old business system and the new business system.
Description
Technical Field
The present invention relates to the field of log processing and analysis technologies, and in particular, to a log data processing method, device and electronic device.
Background
In the enterprise informatization and digitization construction process, according to the actual demands of the current enterprise internal and external various project system construction (established and newly built systems) on log definition, collection and application, the rapid multiplexing and high compatibility of various software system projects are realized through a log classification model, the integration capability of other newly built systems is met, the problem that log management specifications are not uniform finally, log data lookup is not controlled, sensitive data and user behavior information are easy to leak is easily caused, the operation states of an old system and a new system can not be uniformly managed by an operation log, a silent system is found and optimized, and the business supporting capability of the enterprise informatization system is improved.
In the log management and application directions, various log classification methods adopted in the prior art are used for classifying and applying logs in a specific field or type, but are not used for researching logs of old and old systems which are already built and used by enterprises, so that the log management fusion of the old and old systems and the new systems can not be well solved. In addition, in the log collection process, the old system is old due to the technology in construction or does not pay attention to the meaning of the system log to the operation of the service system, and does not output the log with unified specification, so that the following technical problems exist: 1) The old system log is missing, the log data which can be provided or collected is single, and classification management cannot be uniformly performed; 2) Because the new system and the old system adopt different implementation technologies, the generated log data have different specifications, and an effective general log classification method is difficult to establish; 3) Old system log data cannot be effectively applied to an enterprise informatization monitoring operation and maintenance system.
For the above-described problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a method, a device and electronic equipment for processing log data, which at least solve the technical problem that the log data of a new and old business system cannot be effectively classified and managed in the related technology.
According to an aspect of an embodiment of the present invention, there is provided a method for processing log data, including: classifying the log data of the target system based on a pre-constructed general log classification model to obtain system running log data and user operation behavior log data of the target system; acquiring log main body data in the system operation log data, and classifying the system operation log data based on the log main body data to obtain a log main body classification result of the target system; and analyzing the target system according to the log main body classification result of the target system and the log data of the user operation behaviors to determine the running state of the target system.
Optionally, the user operation behavior log data includes: message Id, system code, system name, service version, service code, service name, log level, timestamp, time when a report request was received, time when a log was actually sent to Kafka, user Id, remark, user name, organization code, organization name, user role name, whether it is three-member, traceId, global link tracking Id, current module sub-module, target module, current address, target address, operation in the system, description of operation, whether user business operation is successful, sensitive behavior specification, whether a front end log report of this trigger has a send back end request, if a front end log report of this trigger has a send back end request, whether the back end request is successful, authorization code required for log report interface call, page name, browser model number, user intranet IP, user public network IP city, user local mac address, page open time, page close time, and service release status.
Optionally, the log body classification result includes at least one of: system web request log data, system business process log data, business database access log data, system global exception log data, system activity process log data, and user access log data.
Optionally, the predetermined index includes at least one of: user access index, user operation index, log alarm information statistics display index and application evaluation report information index.
Optionally, when the predetermined index is the user access index, the user access index includes: user activity number, repeated visitor number, daily average function browsing amount, highest daily access function number, function browsing amount increase percentage, current month activity number, total activity days, new visitor, latest visitor, independent visitor, and total visitor.
Optionally, when the predetermined index is the user operation index, the user operation index includes: the function browsing amount, the number of people access functions and the number of access accounts.
Optionally, when the predetermined index is the log alarm information statistics display index, the log alarm information statistics display index includes: abnormal log number, log alarm number, log collection total number, log increment number, alarm system, alarm time and alarm state.
Optionally, when the predetermined index is the application evaluation report information index, the application evaluation report information index includes: system name, operation time, account opening rate, unit coverage, functional effective utilization, active account ratio, system access rate and system access trend.
According to another aspect of the embodiment of the present invention, there is also provided a log data processing apparatus, including: the first processing module is used for classifying the log data of the target system based on a pre-constructed general log classification model to obtain system running log data and user operation behavior log data of the target system; the second processing module is used for acquiring the log main body data in the system operation log data, classifying the system operation log data based on the log main body data, and obtaining a log main body classification result of the target system; and the third processing module is used for analyzing the preset index of the target system according to the log main body classification result of the target system and the log data of the user operation behavior to determine the running state of the target system.
According to another aspect of an embodiment of the present invention, there is also provided an electronic device including a memory and a processor; wherein the memory is for storing one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps of any of the above.
In the embodiment of the invention, the log data of the target system is classified by adopting a general classification model based on a pre-constructed log to obtain the system running log data and the user operation behavior log data of the target system; acquiring log main body data in the system operation log data, and classifying the system operation log data based on the log main body data to obtain a log main body classification result of the target system; and analyzing the target system according to the log main body classification result of the target system and the log data of the user operation behaviors to determine the running state of the target system. That is, the embodiment of the invention uses the general log classification model to classify the log data of the target system to obtain the system running log data and the user operation behavior log data of the target system; then, the log main body data are utilized to classify the system operation log data, and a log main body classification result of the target system is obtained; finally, the log main body classification result of the target system and the log data of the user operation behaviors are utilized to analyze the preset indexes of the target system, and the running state of the target system is determined, so that the technical problem that the log data of the new and old business systems cannot be effectively classified and managed in the related technology is solved, the log data of the new and old business systems are effectively classified and managed, a silent system is effectively found, and the technical effect of meeting the log management fusion requirements of the old business system and the new business system is achieved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:
FIG. 1 is a flowchart of a log data processing method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a log data processing device according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and in the drawings are used for distinguishing between different objects and not for limiting a particular order.
According to an aspect of embodiments of the present invention, there is provided a method of processing log data, it being noted that the steps shown in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.
Fig. 1 is a flowchart of a log data processing method according to an embodiment of the present invention, as shown in fig. 1, where the method includes the following steps:
step S102, classifying the log data of the target system based on a pre-constructed general log classification model to obtain system running log data and user operation behavior log data of the target system;
the target systems include, but are not limited to, old service systems and new service systems; the general log classification model is obtained by training the convolutional neural network by using the feature vector of each system log data in the training data set, wherein the feature vector of each system log data in the training data set is obtained by extracting the feature vector from each system log data in the training data set based on the category of the system log data and the corresponding keyword of the system log data.
Step S104, acquiring log main body data in the system operation log data, and classifying the system operation log data based on the log main body data to obtain a log main body classification result of the target system;
and S106, analyzing the preset index of the target system according to the log main body classification result of the target system and the log data of the user operation behaviors, and determining the running state of the target system.
Through the steps, the log data of the target system can be classified by using the log general classification model, so that the system running log data and the user operation behavior log data of the target system are obtained; then, the log main body data are utilized to classify the system operation log data, and a log main body classification result of the target system is obtained; finally, the log main body classification result of the target system and the log data of the user operation behaviors are utilized to analyze the preset indexes of the target system, and the running state of the target system is determined, so that the technical problem that the log data of the new and old business systems cannot be effectively classified and managed in the related technology is solved, the log data of the new and old business systems are effectively classified and managed, a silent system is effectively found, and the technical effect of meeting the log management fusion requirements of the old business system and the new business system is achieved.
It should be noted that, the method classifies the log data of the target system and analyzes the predetermined index of the target system by using the corresponding classification result, so as to realize the general classification of the user operation and the system operation log, and effectively find the "silent" system according to the operation state of the target system, thereby solving the problem of observability of the operation state of the system.
Optionally, the user operation behavior log data includes, but is not limited to, the following:
message Id (messageId);
systematic coding (systematic id);
system name (systemName);
version of service (version);
service coding (service);
service name (serviceName);
log level (level), FATAL, ERROR, WARN, INFO, BUSSINESS, DEBUG, TRACE (from high to low);
timestamp (timestamp), record log time, yyyy-MM-dd HH: MM: ss.SSS, beijing time;
time of receipt of the report request (receiveTimeStamp), yyyy-MM-dd HH: MM: ss.SSS, beijing time;
the time (sendtime stamp) at which the log was actually sent to Kafka, yyyy-MM-dd HH: MM: ss.sss, beijing time;
user Id (userId);
remarks (notes);
a user name (username);
organization code (orgCode);
organization name (orgName);
user role name (role);
whether it is a three-member role (hasssary), e.g., security administrator-S, system administrator-S, audit administrator-a, type bootean;
the traceId is used for link tracking, the front end generates the identification ID of the request, and the identification ID is carried in the request header x-traceId-header and can adopt UUID;
global link tracking Id (globalTraceId), generated by the front end, records all requests of users in a site access process from opening the site to closing all site pages, and carries the requests in a request header x-global-traceId-header, and UUID can be adopted;
current module (currentModel);
a subModule (subModule) of the current module;
a target module (targetModel);
current address (currentAddr);
a target address (targetAddr);
operation (action) in the system, the reference values of which are: create, read, update, delete, upload, download, log, logo;
an operational description (actionDesc);
whether the user service operation is successful (userActionSuccess), a string, true or false;
sensitive behavior specification (identification);
reporting whether a back-end request (hasSendRequest) is sent or not by the front-end log triggered at the time;
if the front-end log triggered at this time reports that a back-end request is sent, whether the back-end request is successfully invoked (hasRequestSuccessfully);
calling a required authorization code (abst) by a log reporting interface to contact a log center manager for acquisition;
page name (pageName);
browser model (browser model);
a user intranet IP (userIntranetIp), a plurality of IP addresses, and a plurality of segments;
a user public network IP (userPublicIp) for a plurality of IPs and for "splitting;
a user public network IP city (userPublicIptCity);
user local mac address (userlocaaddr);
page open time (PageOpenTime), yyyy-MM-dd HH: MM: ss.SSS, beijing time;
page close time (pageclose time), yyyy-MM-dd HH: MM: ss.SSS, beijing time;
service release status (servicePostStatus), the available values are: development: in research and development; unbuckling: debugging; pilot_run: performing test operation; form_run: and formally operating.
Optionally, the log body classification result includes at least one of the following: system web request log data, system business process log data, business database access log data, system global exception log data, system activity process log data, and user access log data.
When the log body classification result is system network request log data (networkLogDetailInfo), the system network request log data (networkLogDetailInfo) includes, but is not limited to, the following:
message Id (messageId);
log level (level), FATAL, ERROR, WARN, INFO, DEBUG, TRACE (from high to low);
systematic coding (systematic id);
system name (systemName);
service coding (serviceId);
service name (serviceName);
traceId for link tracking;
globalpraceid for global link tracking, all operations logged in from the user to the user log out;
a timestamp (timestamp) recording the log time;
a user Id (userId) from a userCode value set in the access terminal x-user-code-header request header;
an access terminal service process Id (pId);
a user name (userName);
a network request log detail message (networklog detailinfo);
service handling log detail message (servicelog detailinfo);
a data operation log detail message (datalog detailinfo);
global exception log detail message (globalExceptionLogDetaInfo);
user and business sensitive behavior detailed messages (idetification info);
system activity process classification (sysTrailTypeDetailInfo);
service host Ip (hostIp);
service port (serverPort);
service mac address (macAddress);
service hostName (hostName);
service version number (version);
user role information (role), transmitting parameters in an x-user-role-header request header;
whether the user has a three-member character (hasssarrole) and is derived from: x-user-roll-ssa-roll-header, type boolan;
the organization code (orgCode) of the user is transmitted in the request head of the x-user-org-code-header; the organization name (orgName) where the user is located is referred to in the x-user-org-name-header request header.
When the log body classification result is system service processing log data (serviceLogDetailInfo), the system service processing log data (serviceLogDetailInfo) includes, but is not limited to, the following:
operation (action) in the system, the reference values of which are: create, read, update, delete, upload, download, log, logo;
log description (description);
log coding (code);
log information (message);
raw log (rawLog);
fully qualified class name (className);
method name (methodName) (line number);
thread name (threadName);
a return value;
parameter list key-value form (methodParams), parameter name-parameter value;
processing traffic takes time (expendTime).
When the log body classification result is service database access log data (dataLogDetailInfo), the service database access log data (dataLogDetailInfo) includes, but is not limited to, the following:
log coding (code);
log information (message);
raw log (rawLog);
a data source (dataSource);
executed sql statements (sql);
sql execution time consuming (sqlTime);
thread name (threadName);
data Source type (dataSource type): mysql5, mysql8, oracle.
When the log body classification result is system global exception log data (globalExceptionLogDetailInfo), the system global exception log data (globalExceptionLogDetaInfo) includes but is not limited to the following:
log coding (code);
log information (message);
raw log (rawLog);
thread name (threadName);
source IP address (originIp);
request url (requestUrl);
request sources (requestSource), e.g., web/android/ios;
data type (contentType);
request method (requestMethod);
request parameters (requestparameters).
When the log body classification result is system activity process log data (sysTralTypeDetailInfo), the system activity process log data (sysTralTypeDetaInfo) includes, but is not limited to, the following:
service state (servicestatus enum), the reference value of which is: activate: starting activation; running: is running; turn off: closing; crash: crashing;
service release status (servicepoststatus) with reference to: development: in research and development; unbuckling: debugging; pilot_run: performing test operation; form_run: formally running;
when the log body classification result is user access log data (identifier info), the user access log data (identifier info) includes, but is not limited to, the following:
interface call type (type), its reference value is: user: the front end accesses the back end service interface; service: calling system service;
operation type (action), its reference value is: create: creating; read: reading; update: updating; delete: deleting; uplink: uploading; downlink: downloading; login: logging in; logo ut: logging out;
brief description (description).
Optionally, the predetermined index includes at least one of: user access index, user operation index, log alarm information statistics display index and application evaluation report information index.
It should be noted that, in practical application, the predetermined indexes include, but are not limited to, a user access index, a user operation index, a log alert information statistics display index, and an application evaluation report information index.
Optionally, when the predetermined index is a user access index, the user access index includes: user activity number, repeated visitor number, daily average function browsing amount, highest daily access function number, function browsing amount increase percentage, current month activity number, total activity days, new visitor, latest visitor, independent visitor, and total visitor.
Wherein the user activity number represents the number of accounts that have been operated at least once for one month for system login behavior; the repeated visitor number is used for counting the number of users with the number of times of logging in the system being more than 2 in the time period of the day, 7 days and the month respectively; the daily average function browsing amount represents the average daily access amount of the system function, and the total function browsing amount/system running time; the highest daily access function number represents the highest daily access amount of the user access system function up to yesterday, the calculation logic is used for firstly counting daily average access amount, and then the largest daily average access amount is taken out; the function browsing amount increases by a percentage of (month end of current month-month end of last month) -1; the current month active number represents the number of users logged in by the system each month, and repeated visitors are removed; the total active days represent the active days of each user in the application in a statistical period; the new visitor indicates that the first access of a certain cookie is counted as a new visitor in the counting period; the latest visitor means n independent visitors accessing the system in the latest period of time (within 1 hour), and are arranged in reverse order according to the entering time; independent visitors represent the number of visitors accessing the system in one day (0:00-24:00), and each independent internet surfing computer (based on cookie) is regarded as one visitor; the total guests represent the number of guests all accessing the system within the statistics period.
Optionally, when the predetermined index is a user operation index, the user operation index includes: the function browsing amount, the number of people access functions and the number of access accounts.
Wherein, the function browsing amount represents that the click function of the user is recorded as 1 time once in the statistics period; the number of people access functions represents the total amount of function browsing in the counting period/the number of users logged in the system in the counting period (de-duplication); the access account number represents a data source for calculating the number of people's access functions, i.e. the number of users logging into the system in a statistical period (deduplication).
Optionally, when the predetermined index is a log alarm information statistics display index, the log alarm information statistics display index includes: abnormal log number, log alarm number, log collection total number, log increment number, alarm system, alarm time and alarm state.
Wherein the abnormal log number represents the sum of the log number of abnormal events and the error log number; the log alarm number represents the sum of error log numbers; the total log collection number represents the total log amount generated by each business application system; the log increment number represents the log increment of the previous day generated by each business application system; the alarm system represents the name of the system where the alarm event occurs; the alarm time represents the time when the alarm occurs, which is accurate to seconds; the alert status indicates that the alert information is viewed in red font.
Optionally, when the predetermined index is an application evaluation report information index, the application evaluation report information index includes: system name, operation time, account opening rate, unit coverage, functional effective utilization, active account ratio, system access rate and system access trend.
The system name is obtained by identifying each service system;
the operation time is a certain time period selectable through the calendar control;
the account opening rate is 100% of the number of opened accounts/the number of accounts to be opened, wherein the number of the opened accounts represents the number of accounts actually used by the statistics system, only the accounts actually available (normal state) in the system are counted, and the accounts unavailable (forbidden/off-job state) are not counted; the account number to be opened represents the number of system accounts counted based on the position range of the user defined in the construction target when the system stands are promoted;
the unit coverage rate is 100% of the unit number of the used system/the unit number of the system to be used, wherein the unit number of the used system represents the unit number of the personnel of the account in the normal state of actual use of the system grasping; the unit number of the system is used to represent the unit range which is planned to be used in system construction;
the effective utilization rate of the functions is 100% of the number of system functions used by the user/the total number of the system functions, wherein the number of the system menu modules triggered by the user in the statistical period is represented by the number of the system functions used by the user; the total system functions represent the total system menu module statistics; function availability = number of system functions used by the user/total number of system menu modules;
the active account ratio is 100% of the active account number/total account number, wherein the active account number represents the number of users logged in once a month when the system takes place, and the month is a period; the total account number represents the total number of system registered accounts, and the interface obtains the total number of system registered accounts; active account duty cycle = active account number/total account number;
the system access rate is the system month login times/total account number, wherein the system month login times represent the month as a period, and the system month login times are generated; the total account number represents the total number of system registered accounts, and the interface obtains the total number of system registered accounts; system access rate = number of system month logins/total account number;
and the system access change trend is that indexes are displayed in a trend chart, the time dimension is the natural month of the year, and if the current time is 1 month and 1 day, the data of 12 months of the last year are displayed, wherein the normal login times of the system users counted in each month are used as the annual trend chart.
According to another aspect of the embodiment of the present invention, there is further provided a log data processing apparatus, and fig. 2 is a schematic diagram of the log data processing apparatus provided by the embodiment of the present invention, where as shown in fig. 2, the log data processing apparatus includes: a first processing module 22, a second processing module 24, and a third processing module 26. The processing device of the log data will be described in detail below.
The first processing module 22 is configured to perform classification processing on log data of the target system based on a pre-constructed general log classification model, so as to obtain system running log data and user operation behavior log data of the target system;
the second processing module 24 is connected to the first processing module 22, and is configured to obtain log main body data in the system running log data, and perform classification processing on the system running log data based on the log main body data, so as to obtain a log main body classification result of the target system;
the third processing module 26 is connected to the second processing module 24, and is configured to determine an operation state of the target system according to the log body classification result of the target system and the user operation behavior log data by analyzing a predetermined index of the target system.
In the embodiment of the invention, the log data processing device classifies the log data of the target system by using a log general classification model to obtain the system running log data and the user operation behavior log data of the target system; then, the log main body data are utilized to classify the system operation log data, and a log main body classification result of the target system is obtained; finally, the log main body classification result of the target system and the log data of the user operation behaviors are utilized to analyze the preset indexes of the target system, and the running state of the target system is determined, so that the technical problem that the log data of the new and old business systems cannot be effectively classified and managed in the related technology is solved, the log data of the new and old business systems are effectively classified and managed, a silent system is effectively found, and the technical effect of meeting the log management fusion requirements of the old business system and the new business system is achieved.
It should be noted that, the first processing module 22, the second processing module 24, and the third processing module 26 correspond to steps S102 to S106 in the method embodiment, and the foregoing modules are the same as examples and application scenarios implemented by the corresponding steps, but are not limited to those disclosed in the method embodiment.
Optionally, the user operation behavior log data includes: message Id, system code, system name, service version, service code, service name, log level, timestamp, time when a report request was received, time when a log was actually sent to Kafka, user Id, remark, user name, organization code, organization name, user role name, whether it is three-member, traceId, global link tracking Id, current module sub-module, target module, current address, target address, operation in the system, description of operation, whether user business operation is successful, sensitive behavior specification, whether a front end log report of this trigger has a send back end request, if a front end log report of this trigger has a send back end request, whether the back end request is successful, authorization code required for log report interface call, page name, browser model number, user intranet IP, user public network IP city, user local mac address, page open time, page close time, and service release status.
Optionally, the log body classification result includes at least one of the following: system web request log data, system business process log data, business database access log data, system global exception log data, system activity process log data, and user access log data.
Optionally, the predetermined index includes at least one of: user access index, user operation index, log alarm information statistics display index and application evaluation report information index.
Optionally, when the predetermined index is a user access index, the user access index includes: user activity number, repeated visitor number, daily average function browsing amount, highest daily access function number, function browsing amount increase percentage, current month activity number, total activity days, new visitor, latest visitor, independent visitor, and total visitor.
Optionally, when the predetermined index is a user operation index, the user operation index includes: the function browsing amount, the number of people access functions and the number of access accounts.
Optionally, when the predetermined index is a log alarm information statistics display index, the log alarm information statistics display index includes: abnormal log number, log alarm number, log collection total number, log increment number, alarm system, alarm time and alarm state.
Optionally, when the predetermined index is an application evaluation report information index, the application evaluation report information index includes: system name, operation time, account opening rate, unit coverage, functional effective utilization, active account ratio, system access rate and system access trend.
According to another aspect of an embodiment of the present invention, there is also provided an electronic device including a memory and a processor; wherein the memory is for storing one or more computer instructions, wherein the one or more computer instructions are executable by the processor to perform the method steps of any of the above.
The above-described programs may be run on a processor or may also be stored in memory (or referred to as computer-readable media), including both permanent and non-permanent, removable and non-removable media, and information storage may be implemented by any method or technique. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
These computer programs may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks and/or block diagram block or blocks, and corresponding steps may be implemented in different modules.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. As will be apparent to those skilled in the art,
various modifications and changes are possible in the present application. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.
Claims (10)
1. A method for processing log data, comprising:
classifying the log data of the target system based on a pre-constructed general log classification model to obtain system running log data and user operation behavior log data of the target system;
acquiring log main body data in the system operation log data, and classifying the system operation log data based on the log main body data to obtain a log main body classification result of the target system;
and analyzing the target system according to the log main body classification result of the target system and the log data of the user operation behaviors to determine the running state of the target system.
2. The method for processing log data according to claim 1, wherein the user operation behavior log data comprises: message Id, system code, system name, service version, service code, service name, log level, timestamp, time when a report request was received, time when a log was actually sent to Kafka, user Id, remark, user name, organization code, organization name, user role name, whether it is three-member, traceId, global link tracking Id, current module sub-module, target module, current address, target address, operation in the system, description of operation, whether user business operation is successful, sensitive behavior specification, whether a front end log report of this trigger has a send back end request, if a front end log report of this trigger has a send back end request, whether the back end request is successful, authorization code required for log report interface call, page name, browser model number, user intranet IP, user public network IP city, user local mac address, page open time, page close time, and service release status.
3. The method for processing log data according to claim 1, wherein the log body classification result includes at least one of: system web request log data, system business process log data, business database access log data, system global exception log data, system activity process log data, and user access log data.
4. The method of processing log data according to claim 1, wherein the predetermined index includes at least one of: user access index, user operation index, log alarm information statistics display index and application evaluation report information index.
5. The method according to claim 4, wherein when the predetermined index is the user access index, the user access index includes: user activity number, repeated visitor number, daily average function browsing amount, highest daily access function number, function browsing amount increase percentage, current month activity number, total activity days, new visitor, latest visitor, independent visitor, and total visitor.
6. The method according to claim 4, wherein when the predetermined index is the user operation index, the user operation index includes: the function browsing amount, the number of people access functions and the number of access accounts.
7. The method according to claim 4, wherein when the predetermined index is the log alert information statistics display index, the log alert information statistics display index includes: abnormal log number, log alarm number, log collection total number, log increment number, alarm system, alarm time and alarm state.
8. The method according to claim 4, wherein when the predetermined index is the application evaluation report information index, the application evaluation report information index includes: system name, operation time, account opening rate, unit coverage, functional effective utilization, active account ratio, system access rate and system access trend.
9. A log data processing apparatus, comprising:
the first processing module is used for classifying the log data of the target system based on a pre-constructed general log classification model to obtain system running log data and user operation behavior log data of the target system;
the second processing module is used for acquiring the log main body data in the system operation log data, classifying the system operation log data based on the log main body data, and obtaining a log main body classification result of the target system;
and the third processing module is used for analyzing the preset index of the target system according to the log main body classification result of the target system and the log data of the user operation behavior to determine the running state of the target system.
10. An electronic device comprising a memory and a processor; wherein the memory is for storing one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps of any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211682873.6A CN116361127A (en) | 2022-12-27 | 2022-12-27 | Log data processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211682873.6A CN116361127A (en) | 2022-12-27 | 2022-12-27 | Log data processing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116361127A true CN116361127A (en) | 2023-06-30 |
Family
ID=86925956
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211682873.6A Pending CN116361127A (en) | 2022-12-27 | 2022-12-27 | Log data processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116361127A (en) |
-
2022
- 2022-12-27 CN CN202211682873.6A patent/CN116361127A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11188619B2 (en) | Single click delta analysis | |
| US11379275B2 (en) | System and method for tagging and tracking events of an application | |
| US10757546B2 (en) | System and method for triggering on platform usage | |
| CN107832196B (en) | Monitoring device and monitoring method for abnormal content of real-time log | |
| CN110716832B (en) | Service operation monitoring and alarming method, system, electronic equipment and storage medium | |
| US10616254B2 (en) | Data stream surveillance, intelligence and reporting | |
| CN102801785B (en) | System and method for monitoring advertisement putting engine | |
| CN112905548B (en) | Security audit system and method | |
| US11362912B2 (en) | Support ticket platform for improving network infrastructures | |
| CN111131290B (en) | Flow data processing method and device | |
| CN113516337A (en) | Method and device for monitoring data security operation | |
| Winkler | ‘A Unix Prototype for Intrusion and Anomaly Detection in Secure Networks | |
| CN111666205A (en) | Data auditing method, system, computer equipment and storage medium | |
| CN116361127A (en) | Log data processing method and device and electronic equipment | |
| CN115757318A (en) | Log query method and device, storage medium and electronic equipment | |
| CN111259383A (en) | Safety management center system | |
| CN113076230A (en) | Business performance monitoring design method based on Jmeter | |
| Pan | Independent Study of Splunk | |
| CN112905417A (en) | Business performance monitoring design method based on Jmeter | |
| CN115168489A (en) | Data evidence storage method and device based on block chain | |
| CN117194164A (en) | Information integration supervision platform and method based on ESB bus | |
| CN114625643A (en) | Data processing method and device | |
| CN116757662A (en) | Video operation and maintenance management method and platform | |
| CN116028451A (en) | Log analysis method and related equipment | |
| Brhan | API-Based Cloud Data Acquisition and Analysis from Smart Home IoT Environments |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |