CN113672574A - Log processing method, electronic device and computer-readable storage medium - Google Patents

Log processing method, electronic device and computer-readable storage medium Download PDF

Info

Publication number
CN113672574A
CN113672574A CN202110786440.4A CN202110786440A CN113672574A CN 113672574 A CN113672574 A CN 113672574A CN 202110786440 A CN202110786440 A CN 202110786440A CN 113672574 A CN113672574 A CN 113672574A
Authority
CN
China
Prior art keywords
compressed data
event
log
information
abnormal
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110786440.4A
Other languages
Chinese (zh)
Inventor
任鹏
付文杰
申洪涛
付卫东
史轮
陶鹏
吴一敌
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
State Grid Corp of China SGCC
Marketing Service Center of State Grid Hebei Electric Power Co Ltd
Original Assignee
State Grid Corp of China SGCC
Marketing Service Center of State Grid Hebei Electric Power Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by State Grid Corp of China SGCC, Marketing Service Center of State Grid Hebei Electric Power Co Ltd filed Critical State Grid Corp of China SGCC
Priority to CN202110786440.4A priority Critical patent/CN113672574A/en
Publication of CN113672574A publication Critical patent/CN113672574A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/17Details of further file system functions
    • G06F16/174Redundancy elimination performed by the file system
    • G06F16/1744Redundancy elimination performed by the file system using compression, e.g. sparse files
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/1805Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815Journaling file systems

Abstract

The application is applicable to the technical field of electric power, and provides a log processing method, electronic equipment and a computer-readable storage medium. The method comprises the following steps: acquiring a log file, wherein the log file comprises event exception information and event content; compressing the event abnormal information to obtain first compressed data; compressing the event content to obtain second compressed data, wherein the first compressed data and the second compressed data contain the same log identification; and sending the first compressed data and the second compressed data to a server, wherein the first compressed data is used for the server to determine whether the program operation of the terminal equipment is abnormal, and the second compressed data is used for the server to determine the event content corresponding to the abnormality. According to the method and the device, whether the log file is abnormal or not and whether the running of the application program is abnormal or not can be quickly determined.

Description

Log processing method, electronic device and computer-readable storage medium
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a log processing method, an electronic device, and a computer-readable storage medium.
Background
A log refers to a collection of certain operations of objects specified by the system and their results in time order. The log file records information of interaction between the system and users of the system, and is a data collection method for automatically capturing the type, content or time of interaction between a person and a system terminal. The terminal device sends the log file to the server after collecting the log file, and the server can monitor and maintain the system according to the log file. The traditional log processing method has the problems that the processing efficiency is low, so that program abnormity cannot be detected as soon as possible
Disclosure of Invention
In view of this, embodiments of the present application provide a log processing method, an electronic device, and a computer-readable storage medium, which can improve log processing efficiency, so that an application exception can be detected as soon as possible.
In order to achieve the purpose, the technical scheme is as follows:
in a first aspect, an embodiment of the present application provides a method applied to a terminal device, where the log processing method includes:
acquiring a log file, wherein the log file comprises event exception information and event content;
compressing the event abnormal information to obtain first compressed data;
compressing the event content to obtain second compressed data, wherein the first compressed data and the second compressed data contain the same log identification;
and sending the first compressed data and the second compressed data to a server, wherein the first compressed data is used for the server to determine whether the program operation of the terminal equipment is abnormal, and the second compressed data is used for the server to determine the event content corresponding to the abnormality.
In the embodiment of the application, event abnormal information and event content in a log file are respectively compressed to obtain first compressed data and second compressed data, and the first compressed data and the second compressed data are respectively sent to a server. Since the event anomaly information occupies a smaller space and the event content occupies a larger space relative to the event content, respectively sending the first compressed data and the second compressed data to the server enables the event anomaly information to be quickly transmitted to the server instead of transmitting the entire log file to the server. The server can quickly determine whether the operation of the application program is abnormal by detecting the first compressed data, and the detection of the whole log file is not needed. And the first compressed data and the second compressed data corresponding to the same log file have the same log identification, and when the server determines that the event abnormal information represents that the log file is abnormal, the server can determine the corresponding event content through the log identification to acquire detailed abnormal information.
Based on the first aspect, in some embodiments, the compressing the event exception information to obtain first compressed data includes:
converting the event abnormal information into data with preset length;
and compressing the data with the preset length and the log identification to obtain the first compressed data.
Based on the first aspect, in some embodiments, the converting the event exception information into data with a preset length includes:
extracting characteristic information of the event abnormal information, wherein each type of time abnormal information corresponds to one type of the characteristic information;
mapping the characteristic information into an abnormal identifier to obtain data with a preset length; the abnormal marks corresponding to different types of feature information are different, the abnormal marks corresponding to the same type of feature information are the same, and the abnormal marks are represented by data with preset length.
Based on the first aspect, in some embodiments, the sending the first compressed data and the second compressed data to a server includes:
in each data transmission period, if the first compressed data and the second compressed data exist simultaneously and need to be sent to the server, the first compressed data is preferentially sent to the server; alternatively, the first and second electrodes may be,
and in each data transmission period, if the first compressed data does not exist and needs to be sent to the server and the second compressed data exists and needs to be sent to the server, sending the second compressed data to the server.
According to the first aspect, in some embodiments, the log file is a plurality of log files, and each log file corresponds to event exception information and event content;
the compressing the event abnormal information to obtain first compressed data includes:
compressing each event abnormal information respectively to obtain a plurality of first compressed data;
compressing the event content to obtain second compressed data, including:
and compressing each time content respectively to obtain a plurality of second compressed data.
In a second aspect, an embodiment of the present application provides a log processing method, which is applied to a server, and the log processing method includes:
receiving a plurality of first compressed data, wherein each first compressed data comprises event exception information and a log identifier which are in one-to-one correspondence, and the event exception information is part of information in a log file;
receiving a plurality of second compressed data, wherein each second compressed data comprises event content and log identification which are in one-to-one correspondence, and the event content is part of information of a log file;
detecting whether first event abnormal information represents that the log file has an abnormality, wherein the first event abnormal information is any event abnormal information in a plurality of event abnormal information in the plurality of second compressed data;
if the first event abnormal information represents that the log file is abnormal, determining a first log identifier corresponding to the first event abnormal information;
and determining the target event content corresponding to the first event abnormal information according to the matching relation between the first log identifier and the log identifiers in the plurality of second compressed data.
In the embodiment of the application, event abnormal information and event content in a log file are respectively compressed to obtain first compressed data and second compressed data, and the first compressed data and the second compressed data are respectively sent to a server. Since the event anomaly information occupies a smaller space and the event content occupies a larger space relative to the event content, respectively sending the first compressed data and the second compressed data to the server enables the event anomaly information to be quickly transmitted to the server instead of transmitting the entire log file to the server. The server can quickly determine whether the operation of the application program is abnormal by detecting the first compressed data, and the detection of the whole log file is not needed. And the first compressed data and the second compressed data corresponding to the same log file have the same log identification, and when the server determines that the event abnormal information represents that the log file is abnormal, the server can determine the corresponding event content through the log identification to acquire detailed abnormal information.
Based on the second aspect, in some embodiments, the detecting whether the first event exception information characterizes that the log file has an exception includes:
acquiring an abnormal identifier in the first event abnormal information;
and determining whether the log file has abnormity and the abnormity degree according to the abnormity identification.
Based on the second aspect, in some embodiments, the detecting whether the first event exception information characterizes that the log file has an exception includes:
extracting characteristic information of the first event abnormal information;
and determining whether the log file is abnormal or not and the abnormal degree according to the characteristic information.
In a third aspect, an embodiment of the present application provides a log processing apparatus, which is applied to a terminal device, where the log processing apparatus includes:
the system comprises an acquisition module, a storage module and a processing module, wherein the acquisition module is used for acquiring a log file, and the log file comprises event abnormal information and event content;
the first compression module is used for compressing the event abnormal information to obtain first compressed data;
the second compression module is used for compressing the event content to obtain second compressed data, and the first compressed data and the second compressed data contain the same log identifier;
and the sending module is used for sending the first compressed data and the second compressed data to a server, wherein the first compressed data is used for the server to determine whether the program operation of the terminal equipment is abnormal, and the second compressed data is used for the server to determine event content corresponding to the abnormality.
In a fourth aspect, an embodiment of the present application provides a log processing apparatus, which is applied to a server, and the log processing apparatus includes:
the first receiving module is used for receiving a plurality of first compressed data, each first compressed data comprises event exception information and a log identifier which are in one-to-one correspondence, and the event exception information is part of information in a log file;
the second receiving module is used for receiving a plurality of second compressed data, each second compressed data comprises event content and log identification which are in one-to-one correspondence, and the event content is part of information of a log file;
the detection module is used for detecting whether first event abnormal information represents that the log file has abnormality, wherein the first event abnormal information is any event abnormal information in a plurality of event abnormal information in the plurality of second compressed data;
the log identification determining module is used for determining a first log identification corresponding to the first event abnormal information when the first event abnormal information represents that the log file is abnormal;
and the event content determining module is used for determining the target event content corresponding to the first event abnormal information according to the matching relation between the first log identifier and the log identifiers in the plurality of second compressed data.
In a fifth aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the log processing method according to any one of the above first aspects or the steps of the day-to-day processing method according to any one of the above second aspects when executing the computer program.
In a sixth aspect, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the steps of the log processing method according to any one of the first aspect or the second aspect.
In a seventh aspect, an embodiment of the present application provides a computer program product, which, when running on an electronic device, causes the electronic device to execute the steps of the method for diagnosing the fault of the electrical device according to any one of the first aspect, or execute the steps of the log processing method according to any one of the second aspect.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
Fig. 1 is a system architecture diagram of a log processing method provided in an embodiment of the present application;
fig. 2 is a flowchart of a log processing method provided in an embodiment of the present application;
fig. 3 is a scene diagram of a log processing method provided in an embodiment of the present application;
FIG. 4 is a schematic structural diagram of a log processing apparatus according to an embodiment of the present application;
FIG. 5 is a schematic structural diagram of a log processing apparatus according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;
Detailed Description
The present application will be described more clearly with reference to specific examples. The following examples will assist those skilled in the art in further understanding the role of the present application, but are not intended to limit the application in any way. It should be noted that various changes and modifications can be made by one skilled in the art without departing from the spirit of the application. All falling within the scope of protection of the present application.
To make the objects, technical solutions and advantages of the present application more clear, the following description is made by way of specific embodiments with reference to the accompanying drawings.
Fig. 1 shows a system architecture diagram to which the log processing method provided in the embodiment of the present application is applied. Referring to fig. 1, the system architecture may include a server 200 and at least one terminal device 100 (only one shown in fig. 1).
A wireless network connection can be established between the server 200 and the at least one terminal device 100, for example, the wireless network may be Wi-Fi, Bluetooth (BT), LoRa, Zigbee, or the like. Alternatively, a wired network connection can be established between the server 200 and the at least one terminal device 100.
In this embodiment, the system generates a log file when running on the terminal device 100, and the terminal device 100 collects the log file and compresses the log file. After that, the terminal device 100 transmits the compressed log file to the server 200. The server 200 monitors whether the system is abnormal according to the log file, and maintains the system.
By way of example, the terminal device 100 may be a mobile phone, a Personal Digital Assistant (PDA), a tablet computer, a wearable device, an in-vehicle device, an Augmented Reality (AR)/Virtual Reality (VR) device, a portable device (e.g., a laptop computer), an ultra-mobile Personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), a PC (Personal computer), and other terminal devices, which are not limited in this embodiment.
Note that fig. 1 shows only one terminal device 100, but the present application is not limited thereto. In some embodiments, the number of the terminal devices 100 may be two or more, and each terminal device 100 may establish a wireless network connection or a wired network connection with the server 200.
The following describes a log processing method in the embodiment of the present application in detail.
Fig. 2 illustrates a log processing method provided in an embodiment of the present application, which is applicable to the terminal device 100 illustrated in fig. 1. Referring to fig. 2, the log processing method may include steps 101 to 105.
Step 101, obtaining a log file.
For example, event exception information and event content may be included in the log file. The event exception information is used for representing whether the events in the log file are exception events or not and the exception degree. The event content contains information about the event, such as the event occurrence time, the event occurrence location, and event specific information.
The event occurrence time represents the time of event occurrence, for example, when the event occurrence is monitored, the current time is acquired through a network to obtain the event occurrence time. The occurrence position represents a node where the event occurs, and the node may be a certain interface, a certain control, a certain thread, and the like. The event specific information represents the specific content of the event, such as browsing a certain commodity by the user, placing the commodity in a shopping cart by the user, clicking a certain control by the user, and the like.
The event exception information may indicate that an event is an exception or that an event is not an exception. If the event has an abnormality, the abnormality degree can be determined according to the corresponding abnormality name. The abnormality degree may include a plurality of abnormality levels such as a first abnormality level, a second abnormality level, a third abnormality level, a fourth abnormality level, and a fifth abnormality level. The event exception information corresponding to the first exception level may be detailed log information, such as problem diagnosis. The event exception information corresponding to the second exception level may be key node information for confirming whether the application operates as intended. The event exception information corresponding to the third exception level may be information recorded when some unexpected event occurs, for example, the available space of the disk is lower than a preset space, and the application program may run normally. The event exception information corresponding to the fourth exception level may be information recorded that causes some functions of the application program to fail to operate normally for a serious problem. The event exception information corresponding to the fifth exception level may be information recorded when the application program fails to run normally due to a fatal error.
And 102, compressing the event abnormal information to obtain first compressed data.
After a certain log file is obtained, the event exception information of the log file is extracted from the log file. And then, compressing the event abnormal information to obtain first compressed data.
In some embodiments, the event exception information may be converted into data of a predetermined length and then compressed to minimize the time required to transmit the first compressed data to the server. For example, the event exception information is converted into data with a preset length, and then the data with the preset length and the log identifier are compressed to obtain first compressed data. The log identification is a log identification corresponding to a log file where the event abnormal information is located, and each log file corresponds to a unique log identification. That is, for the same log file, the first compressed data in this step has the same log identification as the second compressed data in the following step 104.
In some embodiments, when the event exception information is converted into data with a preset length, the exception degree of the event exception information may be reflected in the data with the preset length.
For example, feature information of event exception information may be extracted, and each type of time exception information corresponds to one type of feature information. And mapping the characteristic information into an abnormal identifier to obtain data with a preset length. The abnormal marks corresponding to different types of feature information are different, the abnormal marks corresponding to the same type of feature information are the same, and the abnormal marks are represented by data with preset length.
As an example, for the first to fifth abnormality levels shown in step 101, the feature information in the event abnormality information corresponding to different abnormality levels is different, and the feature information in the event abnormality information corresponding to the same abnormality level is the same. Therefore, the feature information in the event anomaly information corresponding to the same anomaly level can be mapped to the same anomaly identification, and the feature information in the event anomaly information corresponding to different anomaly levels can be mapped to different anomaly identifications.
For example, feature information in the event anomaly information corresponding to the first anomaly level is mapped to a first anomaly flag, feature information in the event anomaly information corresponding to the second anomaly level is mapped to a second anomaly flag, feature information in the event anomaly information corresponding to the third anomaly level is mapped to a third anomaly flag, feature information in the event anomaly information corresponding to the fourth anomaly level is mapped to a fourth anomaly flag, and feature information in the event anomaly information corresponding to the fifth anomaly level is mapped to a fifth anomaly flag.
In one scenario, the anomaly identifier may be represented by one byte, and different anomaly identifiers have different byte contents. For example, the first to fifth abnormality identifications may be represented by one byte. For example, the first anomaly identification is 1, the second anomaly identification is 2, the third anomaly identification is 3, the fourth anomaly identification is 4, and the fifth anomaly identification is 5.
In another scenario, the anomaly identifier may be represented by two bytes, which are a first byte and a second byte, respectively, where the first byte is used to distinguish different anomaly identifiers, and the second byte is used to carry other information. For example, the first anomaly identification is 1X, the second anomaly identification is 2X, the third anomaly identification is 3X, the fourth anomaly identification is 4X, and the fifth anomaly identification is 5X. 1-5 of the abnormal identifications are used for distinguishing the first abnormal identification from the fifth abnormal identification, and X of the abnormal identifications can be used for bearing other information.
In one scenario, if the degree of abnormality is large, different abnormality identifiers cannot be distinguished by one byte, and different abnormality identifiers can be distinguished by more than two bytes.
It should be noted that the above-mentioned exception flag may also be represented by three or more bytes, which is not limited in this embodiment of the present application.
Step 103, sending the first compressed data to the server.
And step 104, compressing the event content to obtain second compressed data.
After a certain log file is acquired, the event content of the log file is extracted from the log file. And then, compressing the event abnormal information to obtain second compressed data.
Or after a certain log file is acquired, the event exception information of the log file is extracted from the log file. And finally, compressing the residual information of the log file as event content to obtain second compressed data.
Illustratively, step 104 may be: and compressing the event content and the log identification to obtain second compressed data. The log identification is corresponding to the log file where the event abnormal information is located, and each log file corresponds to a unique log identification. That is, for the same log file, the second compressed data in this step has the same log identifier as the first compressed data in step 102.
Step 105, sending the second compressed data to the server.
In some embodiments, step 102 and step 104 may be performed simultaneously for one log file, and step 103 may be performed after step 102 is performed, and step 105 may be performed after step 104 is performed.
For example, after acquiring the log file, the terminal device may simultaneously compress the event exception information and the event content in the log file to obtain first compressed data and second compressed data. After the first compressed data is obtained, the first compressed data can be sent to the server. After the second compressed data is obtained, the second compressed data can be sent to the server.
In still other embodiments, step 102 and step 104 may be performed simultaneously for one log file, step 103 may be performed after step 102 is performed, and step 105 may be performed after step 103 is performed.
For example, after acquiring the log file, the terminal device may simultaneously compress the event exception information and the event content in the log file to obtain first compressed data and second compressed data. After the first compressed data is obtained, the first compressed data can be sent to the server. And after the first compressed data is sent to the server, the second compressed data is sent to the server.
In still other embodiments, for a log file, step 102 may be executed first and then step 104 is executed, step 103 may be executed after step 102 is executed, and step 105 may be executed after step 104 is executed.
For example, after acquiring the log file, the terminal device preferentially compresses the event exception information in the log file to obtain first compressed data. And then, compressing the event content in the log file to obtain second compressed data. After the first compressed content is obtained, the first compressed data can be sent to the server. After the second compressed data is obtained, the second compressed data may be sent to the server.
In the embodiment of the application, event abnormal information and event content in a log file are respectively compressed to obtain first compressed data and second compressed data, and the first compressed data and the second compressed data are respectively sent to a server. Since the event anomaly information occupies a smaller space and the event content occupies a larger space relative to the event content, respectively sending the first compressed data and the second compressed data to the server enables the event anomaly information to be quickly transmitted to the server instead of transmitting the entire log file to the server. The server can quickly determine whether the operation of the application program is abnormal by detecting the first compressed data, and the detection of the whole log file is not needed. And the first compressed data and the second compressed data corresponding to the same log file have the same log identification, and when the server determines that the event abnormal information represents that the log file is abnormal, the server can determine the corresponding event content through the log identification to acquire detailed abnormal information.
In the embodiment of the application, one terminal device can execute the log processing method on one log file or a plurality of log files at the same time. For a plurality of log files, each log file corresponds to event exception information and event content. Step 102 may include: and respectively compressing the abnormal information of each event to obtain a plurality of first compressed data. Step 104 may include: and respectively compressing the event contents to obtain a plurality of second compressed data.
The following describes a process in which the terminal device executes the above-described log processing method on a plurality of log files at the same time, taking log file 1, log files 2 and … …, and log file N (N ≧ 2) as an example.
In a scenario, the terminal device obtains the log file 1, the log file 2, the log file … …, and the log file N, and can process the log file 1, the log file 2, the log file … …, and the log file N according to the process of the log processing method, which is not described herein again.
In another scenario, the terminal device acquires log file 1, log files 2 and … …, and log file N at the same time. Thereafter, the steps 102 and 104 can be respectively performed on the log file 1, the log file 2, … …, and the log file N, resulting in the first compressed data 1, the first compressed data 2, … …, the first compressed data N, and the second compressed data 1, the second compressed data 2, … …, the second compressed data N. After the first compressed data 1, the first compressed data 2, … …, and the first compressed data N are obtained, the first compressed data 1, the first compressed data 2, … …, and the first compressed data N may be transmitted to the server. After the second compressed data 1, the second compressed data 2, … …, and the second compressed data N are obtained, the second compressed data 1, the second compressed data 2, … …, and the second compressed data N may be sent to the server.
In another scenario, the terminal device acquires log file 1, log files 2 and … …, and log file N at the same time. Thereafter, the steps 102 and 104 can be respectively performed on the log file 1, the log file 2, … …, and the log file N, resulting in the first compressed data 1, the first compressed data 2, … …, the first compressed data N, and the second compressed data 1, the second compressed data 2, … …, the second compressed data N. After the first compressed data 1, the first compressed data 2, … …, and the first compressed data N are obtained, the first compressed data 1, the first compressed data 2, … …, and the first compressed data N may be transmitted to the server. After the first compressed data 1, the first compressed data 2, … …, and the first compressed data N are transmitted to the server, the second compressed data 1, the second compressed data 2, … …, and the second compressed data N are transmitted to the server.
In another scenario, the terminal device acquires log file 1, log files 2 and … …, and log file N at the same time. Step 102 is performed on log file 1, log files 2, … …, and log file N to obtain first compressed data 1, first compressed data 2, … …, and first compressed data N. Then, step 104 is performed on the log file 1, the log files 2 and … …, and the log file N to obtain second compressed data 1, second compressed data 2 and … …, and second compressed data N. After the first compressed data 1, the first compressed data 2, … …, and the first compressed data N are obtained, the first compressed data 1, the first compressed data 2, … …, and the first compressed data N may be transmitted to the server. After the second compressed data 1, the second compressed data 2, … …, and the second compressed data N are obtained, the second compressed data 1, the second compressed data 2, … …, and the second compressed data N may be transmitted to the server.
In some embodiments, the obtained plurality of first compressed data may be written into the first queue, and each first compressed data may be sent to the server according to a first-in first-out rule.
In some embodiments, the obtained plurality of second compressed data may be written into the second queue, and each second compressed data may be sent to the server according to a first-in first-out rule.
In some embodiments, in each data transmission period, if the first compressed data and the second compressed data exist simultaneously and need to be sent to the server, the first compressed data is preferentially sent to the server.
For example, in one data transmission period, it is detected whether there is first compressed data and second compressed data to be sent to the server. If the first compressed data 1 and the second compressed data 1 need to be sent to the server, the first compressed data 1 is preferentially sent to the server in the current data transmission period. And continuously detecting whether the first compressed data and the second compressed data need to be sent to the server or not in the next data transmission period. If the first compressed data 2 and the second compressed data 1 need to be sent to the server, the first compressed data 2 is preferentially sent to the server in the current data transmission period.
For another example, in one data transmission period, whether the first compressed data and the second compressed data need to be sent to the server is detected. If the first compressed data 1, the first compressed data 2 and … …, the first compressed data N, the second compressed data 1, the second compressed data 2 and … … and the second compressed data N need to be sent to the server, the first compressed data are preferentially sent to the server in the current data transmission period. Specifically, the first compressed data before may be selected to be sent to the server according to the order of the first compressed data 1, the first compressed data 2, … …, and the first compressed data N in the first queue.
In some embodiments, in each data transmission period, if there is no first compressed data to be sent to the server and there is a second compressed data to be sent to the server, the second compressed data is sent to the server.
For example, in one data transmission period, it is detected whether there is first compressed data and second compressed data to be sent to the server. And if the first compressed data does not exist and needs to be sent to the server and the second compressed data 1 needs to be sent to the server, sending the second compressed data 1 to the server in the current data transmission period. And continuously detecting whether the first compressed data and the second compressed data need to be sent to the server or not in the next data transmission period. And if the first compressed data does not exist and needs to be sent to the server and the second compressed data 2 needs to be sent to the server, sending the second compressed data 2 to the server in the current data transmission period.
For another example, in one data transmission period, whether the first compressed data and the second compressed data need to be sent to the server is detected. And if the first compressed data does not exist and needs to be sent to the server, the second compressed data 1, the second compressed data 2 and … … exist and the second compressed data N needs to be sent to the server, and the second compressed data is sent to the server in the current data transmission period. Specifically, according to the sequence of the second compressed data 1, the second compressed data 2, … …, and the second compressed data N in the second queue, the second compressed data before the second compressed data N is selected to be sent to the server, so that when the server detects that some event abnormal information represents that the program operates abnormally, the server can obtain the corresponding event content as soon as possible to view the specific abnormal information to maintain the application program.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
Referring to fig. 3, an embodiment of the present application provides a log processing method, which is applied to the server shown in fig. 1. Referring to fig. 3, the log processing method may include steps 201 to 205.
Step 201, a plurality of first compressed data are received.
Each first compressed data comprises event abnormal information and log identification which are in one-to-one correspondence, and the event abnormal information is part of information in a log file.
At step 202, a plurality of second compressed data are received.
Each second compressed data comprises event content and log identification which are in one-to-one correspondence, and the event content is part of information of a log file.
Step 203, detecting whether the first event exception information represents that the log file has an exception.
The first event exception information is any one of a plurality of event exception information in the plurality of second compressed data. The log file exception may be whether the running of the application program is daily.
In some embodiments, step 203 may comprise: and acquiring an abnormal identifier in the abnormal information of the first event, and determining whether the log file is abnormal or not and the abnormal degree according to the abnormal identifier. In which the correspondence between the abnormality flag, whether there is an abnormality, and the degree of the abnormality may be set in advance. Please refer to the contents in steps 101 to 105 for the related contents of the abnormal flag and the abnormal degree, which are not described herein again.
In some embodiments, step 203 may comprise: and extracting the characteristic information of the abnormal information of the first event, and determining whether the log file has abnormality and the abnormal degree according to the characteristic information. The characteristic information may be information capable of characterizing whether the log file has an abnormality and an abnormality degree. And the characteristic information may be information inherent in the log file without setting conversion.
Step 204, if the first event abnormal information represents that the log file has an abnormality, determining a first log identifier corresponding to the first event abnormal information.
If the first event abnormality information represents that the log file has an abnormality, the log identifier in the corresponding first compressed data can be determined as the first log identifier.
First compressed data 1, first compressed data 2, … …, and first compressed data N will be described as an example. The first compressed data 1 comprises event exception information 1 and a log identifier 11, the first compressed data 2 comprises event exception information 2 and a log identifier 12, and the first compressed data N comprises event exception information N and a log identifier 1N. And if the server detects that the log file represented by the event abnormal information 2 has an abnormality, taking the log identifier 12 as a first log identifier.
And step 205, determining the target event content corresponding to the first event abnormal information according to the matching relation between the first log identifier and the log identifiers in the plurality of second compressed data.
Illustratively, the first compressed data 1 contains event exception information 1 and a log identifier 11, the first compressed data 2 contains event exception information 2 and a log identifier 12, … …, and the first compressed data N contains event exception information N and a log identifier 1N. The second compressed data 1 includes the event content 1 and the log identifier 21, the second compressed data 2 includes the event content 2 and the log identifiers 22 and … …, and the second compressed data N includes the event content 3 and the log identifier 31. Note that log id 11 is the same as log id 21, log id 12 is the same as log id 22, … …, and log id 1N is the same as log id 2N.
And if the server detects that the log file represented by the event abnormal information 2 has an abnormality, taking the log identifier 12 as a first log identifier. After that, the event content 2 corresponding to the log flag 22 identical to the log flag 12 is set as the target event content corresponding to the first event abnormality information. After obtaining the event content 2, the server determines the abnormal event occurrence time, the abnormal event occurrence position and the specific information of the abnormal event according to the event content 2.
In this embodiment, when a terminal device sends the first compressed data and/or the second compressed data to the server, the server may perform step 201 and/or step 202. After receiving the first compressed data, the server may execute step 203 to detect whether the event exception information in the first compressed data represents that the log file has an exception. If the log file has an exception, step 204 and step 205 are executed to obtain the corresponding event content.
According to the log processing method, after the server receives the first compressed data, the server can detect the event abnormal information in the first compressed data, so that whether the log file is abnormal or not and whether the operation of the application program is abnormal or not can be quickly determined. And after the abnormity is determined, the corresponding event content can be obtained according to the log marks in the first compressed data and the second compressed data.
Referring to fig. 4, an embodiment of the present application provides a log processing apparatus 300, which may be applied to the terminal device 100 shown in fig. 1. The log processing apparatus 300 may include: an obtaining module 301, a first compressing module 302, a second compressing module 303 and a sending module 304.
The obtaining module 301 is configured to obtain a log file, where the log file includes event exception information and event content. The first compression module 302 is configured to compress the event exception information to obtain first compressed data. A second compression module 303, configured to compress the event content to obtain second compressed data, where the first compressed data and the second compressed data include the same log identifier. A sending module 304, configured to send the first compressed data and the second compressed data to a server, where the first compressed data is used by the server to determine whether the program operation of the terminal device is abnormal, and the second compressed data is used by the server to determine event content corresponding to the abnormality.
Referring to fig. 5, an embodiment of the present application provides a log processing apparatus 400, which may be applied to the server 200 shown in fig. 1. The log processing apparatus 400 may include: a first receiving module 401, a second receiving module 402, a detecting module 403, a log identification determining module 404, and an event content determining module 405.
The first receiving module 401 is configured to receive a plurality of first compressed data, where each of the first compressed data includes event exception information and a log identifier that correspond to each other one to one, and the event exception information is a part of information in a log file.
A second receiving module 402, configured to receive multiple pieces of second compressed data, where each piece of the second compressed data includes event content and a log identifier that correspond to each other one to one, and the event content is a part of information of a log file.
A detecting module 403, configured to detect whether first event exception information represents that the log file has an exception, where the first event exception information is any event exception information in a plurality of event exception information in the plurality of second compressed data.
A log identifier determining module 404, configured to determine, when the first event exception information represents that the log file is abnormal, a first log identifier corresponding to the first event exception information.
An event content determining module 405, configured to determine, according to a matching relationship between the first log identifier and the log identifiers in the plurality of second compressed data, a target event content corresponding to the first event exception information.
Fig. 6 is a schematic diagram of an electronic device according to an embodiment of the present invention. As shown in fig. 6, the electronic apparatus 500 of this embodiment includes: a processor 501, a memory 502 and a computer program 503, such as a log processing program, stored in said memory 502 and executable on said processor 501. The processor 501 executes the computer program 503 to implement the steps in the above-mentioned log processing method embodiment, such as steps 101 to 105 shown in fig. 2 or steps 201 to 205 shown in fig. 3. Alternatively, the processor 501, when executing the computer program 503, implements the functions of each module/unit in the above device embodiments, for example, the functions of the modules 301 to 304 shown in fig. 4 or the functions of the modules 401 to 405 shown in fig. 5.
Illustratively, the computer program 503 may be partitioned into one or more modules/units that are stored in the memory 502 and executed by the processor 501 to implement the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 503 in the electronic device 500. For example, the computer program 503 may be divided into the modules shown in fig. 4, or the modules shown in fig. 5.
The electronic device 500 may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The electronic device may include, but is not limited to, a processor 501, a memory 502. Those skilled in the art will appreciate that fig. 5 is merely an example of an electronic device 500 and does not constitute a limitation of electronic device 500 and may include more or fewer components than shown, or some components may be combined, or different components, e.g., the electronic device may also include input-output devices, network access devices, buses, etc.
The Processor 501 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The storage 502 may be an internal storage unit of the electronic device 500, such as a hard disk or a memory of the electronic device 500. The memory 502 may also be an external storage device of the electronic device 500, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device 500. Further, the memory 502 may also include both internal storage units and external storage devices of the electronic device 500. The memory 502 is used for storing the computer programs and other programs and data required by the electronic device. The memory 502 may also be used to temporarily store data that has been output or is to be output.
In some embodiments, the electronic device 500 may be the terminal device shown in fig. 1, or may be the server shown in fig. 1.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments may be implemented. . Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.

Claims (10)

1. A log processing method is applied to terminal equipment and comprises the following steps:
acquiring a log file, wherein the log file comprises event exception information and event content;
compressing the event abnormal information to obtain first compressed data;
compressing the event content to obtain second compressed data, wherein the first compressed data and the second compressed data contain the same log identification;
and sending the first compressed data and the second compressed data to a server, wherein the first compressed data is used for the server to determine whether the program operation of the terminal equipment is abnormal, and the second compressed data is used for the server to determine the event content corresponding to the abnormality.
2. The log processing method according to claim 1, wherein the compressing the event exception information to obtain first compressed data comprises:
converting the event abnormal information into data with preset length;
and compressing the data with the preset length and the log identification to obtain the first compressed data.
3. The log processing method according to claim 2, wherein the converting the event exception information into data of a preset length includes:
extracting characteristic information of the event abnormal information, wherein each type of time abnormal information corresponds to one type of the characteristic information;
mapping the characteristic information into an abnormal identifier to obtain data with a preset length; the abnormal marks corresponding to different types of feature information are different, the abnormal marks corresponding to the same type of feature information are the same, and the abnormal marks are represented by data with preset length.
4. The log processing method according to any one of claims 1 to 3, wherein the sending the first compressed data and the second compressed data to a server includes:
in each data transmission period, if the first compressed data and the second compressed data exist simultaneously and need to be sent to the server, the first compressed data is preferentially sent to the server; alternatively, the first and second electrodes may be,
and in each data transmission period, if the first compressed data does not exist and needs to be sent to the server and the second compressed data exists and needs to be sent to the server, sending the second compressed data to the server.
5. The log processing method according to any one of claims 1 to 3, wherein the log file is plural, and each of the log files corresponds to an event exception information and an event content;
the compressing the event abnormal information to obtain first compressed data includes:
compressing each event abnormal information respectively to obtain a plurality of first compressed data;
compressing the event content to obtain second compressed data, including:
and respectively compressing the event contents to obtain a plurality of second compressed data.
6. A log processing method is applied to a server, and comprises the following steps:
receiving a plurality of first compressed data, wherein each first compressed data comprises event exception information and a log identifier which are in one-to-one correspondence, and the event exception information is part of information in a log file;
receiving a plurality of second compressed data, wherein each second compressed data comprises event content and log identification which are in one-to-one correspondence, and the event content is part of information of a log file;
detecting whether first event abnormal information represents that the log file has an abnormality, wherein the first event abnormal information is any event abnormal information in a plurality of event abnormal information in the plurality of second compressed data;
if the first event abnormal information represents that the log file is abnormal, determining a first log identifier corresponding to the first event abnormal information;
and determining the target event content corresponding to the first event abnormal information according to the matching relation between the first log identifier and the log identifiers in the plurality of second compressed data.
7. The log processing method as claimed in claim 6, wherein said detecting whether the first event exception information characterizes that there is an exception in the log file comprises:
acquiring an abnormal identifier in the first event abnormal information;
and determining whether the log file has abnormity and the abnormity degree according to the abnormity identification.
8. The log processing method as claimed in claim 6, wherein said detecting whether the first event exception information characterizes that there is an exception in the log file comprises:
extracting characteristic information of the first event abnormal information;
and determining whether the log file is abnormal or not and the abnormal degree according to the characteristic information.
9. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the log processing method as claimed in any one of claims 1 to 5 above or implements the steps of the log processing method as claimed in any one of claims 6 to 8 above when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out the steps of the log processing method according to any one of claims 1 to 5 above or the steps of the log processing method according to any one of claims 6 to 8 above.
CN202110786440.4A 2021-07-12 2021-07-12 Log processing method, electronic device and computer-readable storage medium Pending CN113672574A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110786440.4A CN113672574A (en) 2021-07-12 2021-07-12 Log processing method, electronic device and computer-readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110786440.4A CN113672574A (en) 2021-07-12 2021-07-12 Log processing method, electronic device and computer-readable storage medium

Publications (1)

Publication Number Publication Date
CN113672574A true CN113672574A (en) 2021-11-19

Family

ID=78538972

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110786440.4A Pending CN113672574A (en) 2021-07-12 2021-07-12 Log processing method, electronic device and computer-readable storage medium

Country Status (1)

Country Link
CN (1) CN113672574A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023240993A1 (en) * 2022-06-14 2023-12-21 Oppo广东移动通信有限公司 Abnormal positioning solving method and related apparatus

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104572411A (en) * 2013-10-10 2015-04-29 腾讯科技(深圳)有限公司 Log data processing method and device
CN105703932A (en) * 2014-11-28 2016-06-22 广州华多网络科技有限公司 Log recording and uploading method, log recording and reception method, terminal, and server
CN106055465A (en) * 2016-05-31 2016-10-26 广东能龙教育股份有限公司 Client error abnormal information statistical method
CN108011966A (en) * 2017-12-14 2018-05-08 广东金赋科技股份有限公司 The optimization method that a kind of self-aided terminal log compression uploads
CN109240886A (en) * 2018-09-03 2019-01-18 平安科技(深圳)有限公司 Abnormality eliminating method, device, computer equipment and storage medium
US10210162B1 (en) * 2010-03-29 2019-02-19 Carbonite, Inc. Log file management
US20190286628A1 (en) * 2016-12-08 2019-09-19 Zhejiang Dahua Technology Co., Ltd. Methods and systems for processing log data
CN111125042A (en) * 2019-11-13 2020-05-08 中国建设银行股份有限公司 Method and device for determining risk operation event
CN112148518A (en) * 2020-11-02 2020-12-29 深圳市九九互动科技有限公司 Log file processing method and device, computer equipment and storage medium
CN112416634A (en) * 2019-08-22 2021-02-26 中移(苏州)软件技术有限公司 File processing method and device and storage medium
US20210149790A1 (en) * 2019-11-20 2021-05-20 International Business Machines Corporation Event specific log file generation
CN112988682A (en) * 2019-12-18 2021-06-18 阿里巴巴集团控股有限公司 Log compression method and device, electronic equipment and computer readable storage medium

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10210162B1 (en) * 2010-03-29 2019-02-19 Carbonite, Inc. Log file management
CN104572411A (en) * 2013-10-10 2015-04-29 腾讯科技(深圳)有限公司 Log data processing method and device
CN105703932A (en) * 2014-11-28 2016-06-22 广州华多网络科技有限公司 Log recording and uploading method, log recording and reception method, terminal, and server
CN106055465A (en) * 2016-05-31 2016-10-26 广东能龙教育股份有限公司 Client error abnormal information statistical method
US20190286628A1 (en) * 2016-12-08 2019-09-19 Zhejiang Dahua Technology Co., Ltd. Methods and systems for processing log data
CN108011966A (en) * 2017-12-14 2018-05-08 广东金赋科技股份有限公司 The optimization method that a kind of self-aided terminal log compression uploads
CN109240886A (en) * 2018-09-03 2019-01-18 平安科技(深圳)有限公司 Abnormality eliminating method, device, computer equipment and storage medium
CN112416634A (en) * 2019-08-22 2021-02-26 中移(苏州)软件技术有限公司 File processing method and device and storage medium
CN111125042A (en) * 2019-11-13 2020-05-08 中国建设银行股份有限公司 Method and device for determining risk operation event
US20210149790A1 (en) * 2019-11-20 2021-05-20 International Business Machines Corporation Event specific log file generation
CN112988682A (en) * 2019-12-18 2021-06-18 阿里巴巴集团控股有限公司 Log compression method and device, electronic equipment and computer readable storage medium
CN112148518A (en) * 2020-11-02 2020-12-29 深圳市九九互动科技有限公司 Log file processing method and device, computer equipment and storage medium

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023240993A1 (en) * 2022-06-14 2023-12-21 Oppo广东移动通信有限公司 Abnormal positioning solving method and related apparatus

Similar Documents

Publication Publication Date Title
CN111831548B (en) Dependency relationship topological graph drawing method and device
CN108833459B (en) Data reporting method and terminal equipment
CN111078140B (en) Nuclear power station file uploading management method and device, terminal equipment and medium
CN108881454B (en) File transmission method, mobile terminal and storage medium
EP4062293A1 (en) Method and apparatus for storing data, and computer device and storage medium thereof
CN112017323A (en) Patrol alarm method and device, readable storage medium and terminal equipment
WO2021068891A1 (en) Method, system, electronic device, and storage medium for storing and collecting temperature data
CN113672574A (en) Log processing method, electronic device and computer-readable storage medium
CN111130944B (en) System monitoring method and system
CN113506045A (en) Risk user identification method, device, equipment and medium based on mobile equipment
CN111680104A (en) Data synchronization method and device, computer equipment and readable storage medium
CN110851334A (en) Flow statistical method, electronic device, system and medium
US20070282920A1 (en) Job management system and method for minimizing log data
CN113794725A (en) Electric digital data transmission method and system
CN111654405B (en) Method, device, equipment and storage medium for fault node of communication link
WO2022110405A1 (en) Blockchain-based on-chain storage method and apparatus, terminal device, and medium
CN111198916B (en) Data transmission method and device, electronic equipment and storage medium
CN108763053B (en) Method for generating buried point name and terminal equipment
CN113110965A (en) Abnormal information monitoring method and device, computer storage medium and terminal
CN112579406A (en) Log call chain generation method and device
CN113434604B (en) Time sequence database synchronization method, system, equipment and storage medium
CN114691723A (en) Industrial data processing method, device, equipment and medium
CN115905119A (en) BMC log compression and extraction method and system
CN114531340A (en) Log acquisition method and device, electronic equipment, chip and storage medium
CN113094088A (en) Database configuration information acquisition method and device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination