CN111309696A - Log processing method and device, electronic equipment and readable medium - Google Patents

Log processing method and device, electronic equipment and readable medium Download PDF

Info

Publication number
CN111309696A
CN111309696A CN202010200916.7A CN202010200916A CN111309696A CN 111309696 A CN111309696 A CN 111309696A CN 202010200916 A CN202010200916 A CN 202010200916A CN 111309696 A CN111309696 A CN 111309696A
Authority
CN
China
Prior art keywords
log
unresolved
log file
processing
file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010200916.7A
Other languages
Chinese (zh)
Inventor
张祖慧
戴海燕
张权
刘鹏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Cec Cyberspace Great Wall Co ltd
Original Assignee
Cec Cyberspace Great Wall Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Cec Cyberspace Great Wall Co ltd filed Critical Cec Cyberspace Great Wall Co ltd
Priority to CN202010200916.7A priority Critical patent/CN111309696A/en
Publication of CN111309696A publication Critical patent/CN111309696A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/13—File access structures, e.g. distributed indices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/162—Delete operations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/17—Details of further file system functions
    • G06F16/1727—Details of free space management performed by the file system

Abstract

The invention discloses a log processing method and device, electronic equipment and a readable medium, wherein the method comprises the following steps: acquiring an unresolved log file and attribute information of terminal equipment corresponding to the unresolved log file; judging whether the unresolved log file meets the sending condition or not; if so, generating and sending a log reporting message to the management device according to the unresolved log file and the attribute information of the terminal equipment, so that the management device processes the unresolved log file according to a log management strategy; otherwise, storing the unanalyzed log file to the local, and processing the unanalyzed log file according to the storage time of the unanalyzed log file. The method has the advantages that the unanalyzed log files meeting the sending conditions can be sent to the management device for processing, the processing pressure of the log processing device is reduced, and the processing efficiency of the accumulated log information is improved; meanwhile, the log files stored locally can be processed in time, the occupied condition of the storage space is reduced, and the efficiency of log analysis is improved.

Description

Log processing method and device, electronic equipment and readable medium
Technical Field
The invention relates to the technical field of computers, in particular to a log processing method and device, electronic equipment and a readable medium.
Background
At present, with the increasing of terminal devices, more and more log information of each terminal device is collected by a log processing device, and the processing rules of different types of log information are different. The log processing device needs to analyze the logs and then upload the logs to the management device for log management.
If the number of the log information acquired by the log processing device exceeds the processing capacity of the log processing device, the log information is temporarily accumulated in the log processing device, so that the storage space of the log processing device is occupied and the log information cannot be analyzed in time; if the log processing device cannot process the accumulated log information in time and reports the unanalyzed log information to the management device, the system performance of the management device may be degraded.
Disclosure of Invention
Therefore, the invention provides a log processing method and device, electronic equipment and a readable medium, which aim to solve the problems that in the prior art, a log processing device cannot process accumulated log information in time, so that the storage space is occupied and the log information cannot be analyzed in time.
In order to achieve the above object, a first aspect of the present invention provides a log processing method, including: acquiring an unresolved log file and attribute information of terminal equipment corresponding to the unresolved log file; judging whether the unresolved log file meets the sending condition or not; if so, generating and sending a log reporting message to the management device according to the unresolved log file and the attribute information of the terminal equipment, so that the management device processes the unresolved log file according to a log management strategy; otherwise, storing the unanalyzed log file to the local, and processing the unanalyzed log file according to the storage time of the unanalyzed log file.
In some embodiments, saving the unresolved log file to a local location and processing the unresolved log file according to a storage time of the unresolved log file includes: storing the unanalyzed log file into a local first preset folder, wherein the first preset folder is a folder established according to the attribute information of the terminal equipment; and deleting the unanalyzed log file at intervals of a first preset time according to the storage time of the unanalyzed log file, the storage capacity of the first preset folder and the size of the unanalyzed log file.
In some embodiments, deleting the unresolved log file according to a storage time of the unresolved log file, a storage capacity of a first preset folder, and a size of the unresolved log file includes: monitoring the storage capacity of a first preset folder; comparing the storage capacity of the first preset folder with a first storage threshold value; if the storage capacity of the first preset folder is determined to be larger than a first storage threshold, calculating the number of first-class files to be deleted according to the size of the unresolved log files, the first storage threshold and the storage capacity of the first preset folder, wherein the number of the first-class files to be deleted is the calculated number of the unresolved log files to be deleted; sequencing the unresolved log files stored in a first preset folder according to the storage time of the unresolved log files to obtain a first sequencing result; and deleting the unresolved log files according to the number of the first type of files to be deleted and the first sequencing result.
In some embodiments, before the step of obtaining the unresolved log file and the attribute information of the terminal device corresponding to the unresolved log file, the method further includes: the processing rule is obtained in response to a configuration message sent by the management device, wherein the processing rule is a rule verified by the management device.
In some embodiments, obtaining an unresolved log file comprises: collecting log information reported by terminal equipment; judging whether the log information can be processed by using the processing rule; if it is determined that the log information cannot be processed using the processing rule, the log information is saved to an unresolved log file.
In some embodiments, the transmission conditions include: the current moment is in a preset sending time period, and the number of sent pieces in the preset sending time period is smaller than a preset threshold of the number of sent pieces.
In some embodiments, the attribute information of the terminal device includes at least any one of an identifier of the terminal device, type information of the terminal device, and address information of the terminal device.
In order to achieve the above object, a second aspect of the present invention provides a log processing method, including: responding to a log reporting message sent by a log processing device, and acquiring an unresolved log file; and processing the unanalyzed log file according to the log management strategy.
In some embodiments, processing the unresolved log file according to a log management policy includes: establishing a second preset folder according to the address information of the log processing device; storing the unresolved log file into a second preset folder; and deleting the unanalyzed log file at every interval of a second preset time according to the storage capacity of the second preset folder and the log management strategy.
In some embodiments, deleting the unresolved log file according to the storage capacity of the second preset folder and the log management policy includes: monitoring the storage capacity of a second preset folder; comparing the storage capacity of the second preset folder with a second storage threshold; if the storage capacity of the second preset folder is determined to be larger than a second storage threshold, calculating the number of second files to be deleted according to the size of the unresolved log files, the second storage threshold and the storage capacity of the second preset folder, wherein the number of the second files to be deleted is the number of the unresolved log files to be deleted; sequencing the unanalyzed log files stored in the second preset folder according to the storage time of the unanalyzed log files to obtain a second sequencing result; and deleting the unresolved log files according to the second sequencing result and the number of the second type files needing to be deleted.
In some embodiments, before the step of obtaining the unresolved log file in response to a log reporting message sent by the log processing apparatus, the method further includes: acquiring a processing rule; judging whether the processing rule conforms to the rule writing specification; and if the processing rule is determined to meet the rule writing specification, generating and sending a configuration message to the log processing device according to the processing rule so that the log processing device can acquire the processing rule, and processing the log information reported by the terminal equipment according to the processing rule to acquire an unresolved log file.
In some embodiments, processing the unresolved log file according to a log management policy further includes: and deleting the unresolved log file corresponding to the log processing device.
In order to achieve the above object, a third aspect of the present invention provides a log processing apparatus comprising: the first acquisition module is used for acquiring the unanalyzed log file and the attribute information of the terminal equipment corresponding to the unanalyzed log file; the judging module is used for judging whether the unanalyzed log file meets the sending condition or not and obtaining a judging result; the sending module is used for generating and sending a log reporting message to the management device according to the unresolved log file and the attribute information of the terminal equipment when the unresolved log file meets the sending condition according to the judgment result so that the management device can process the unresolved log file according to the log management strategy; and the first processing module is used for storing the unanalyzed log file to the local and processing the unanalyzed log file according to the storage time of the unanalyzed log file when the judgment result shows that the unanalyzed log file does not meet the sending condition.
In order to achieve the above object, a fourth aspect of the present invention provides a management apparatus comprising: the second acquisition module is used for responding to the log reporting message sent by the log processing device and acquiring the unresolved log file; and the second processing module is used for processing the unanalyzed log file according to the log management strategy.
In order to achieve the above object, a fifth aspect of the present invention provides an electronic apparatus comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method of the first aspect or the method of the second aspect.
In order to achieve the above object, a sixth aspect of the present invention provides a computer-readable medium on which a computer program is stored, the program, when executed by a processor, implementing the method of the first aspect, or the method of the second aspect.
The invention has the following advantages: judging whether the obtained unanalyzed log file meets a sending condition, if so, generating and sending a log reporting message to a management device according to the unanalyzed log file and the attribute information of the terminal equipment corresponding to the unanalyzed log file, so that the management device processes the unanalyzed log file according to a log management strategy; the method has the advantages that the unanalyzed log files meeting the sending conditions can be sent to the management device for processing, the processing pressure of the log processing device is reduced, and the processing efficiency of the accumulated log information is improved; if the unanalyzed log file does not meet the sending condition, the unanalyzed log file is stored to the local, and the unanalyzed log file is processed according to the storage time of the unanalyzed log file, so that the residual unanalyzed log file in the accumulated log information can be processed in time, the condition that the storage space is occupied is reduced, and the efficiency of log analysis is improved.
Drawings
The accompanying drawings are included to provide a further understanding of the embodiments of the disclosure and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the principles of the disclosure and not to limit the disclosure. The above and other features and advantages will become more apparent to those skilled in the art by describing in detail exemplary embodiments thereof with reference to the attached drawings, in which:
fig. 1 is a flowchart of a log processing method according to a first embodiment of the present application.
Fig. 2 is a flowchart of a log processing method in the second embodiment of the present application.
Fig. 3 is a flowchart of a log processing method in the third embodiment of the present application.
Fig. 4 is a flowchart of a log processing method in the fourth embodiment of the present application.
Fig. 5 is a schematic structural diagram of a log processing apparatus according to a fifth embodiment of the present application.
Fig. 6 is a schematic structural diagram of a management device according to a sixth embodiment of the present application.
Fig. 7 is a configuration diagram of a log processing system in a seventh embodiment of the present application.
Fig. 8 is a flowchart of a working method of the log processing system in the seventh embodiment of the present application.
Fig. 9 is a block diagram of an exemplary hardware architecture of an electronic device in an eighth embodiment of the present application, where the electronic device can implement a log processing method according to the embodiment of the present application.
Detailed Description
The following detailed description of embodiments of the present application will be made with reference to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present application, are given by way of illustration and explanation only, and are not intended to limit the present application. It will be apparent to one skilled in the art that the present application may be practiced without some of these specific details. The following description of the embodiments is merely intended to provide a better understanding of the present application by illustrating examples thereof.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
Example one
The embodiment of the application provides a log processing method which can be applied to a log processing device. Fig. 1 is a flowchart of a log processing method in this embodiment, which includes:
and step 110, acquiring the unanalyzed log file and the attribute information of the terminal equipment corresponding to the unanalyzed log file.
Specifically, after the terminal device is connected to the log processing device, the log information generated by the terminal device needs to be reported to the log processing device, so that the log processing device can process the log information of the terminal device, and store the log information into the corresponding unresolved log file according to the information such as the type of the terminal device and the type of the log information, so as to facilitate a subsequent log processing device to parse the unresolved log file and obtain useful information therein.
In some embodiments, the attribute information of the terminal device includes at least any one of an identifier of the terminal device, type information of the terminal device, and address information of the terminal device. For example, the device identification number (ID) of a terminal device is 105, the device type of the terminal device is a smartphone, and the Internet Protocol address (IP) of the terminal device is 192.168.1.25.
Step 120, determining whether the unanalyzed log file meets the sending condition.
It should be noted that the log processing device has a limited processing capability for the log information reported by each terminal device, but if all collected unanalyzed log files are uploaded to the management device, the log processing device occupies too much storage space of the management device, and further affects the performance of the management device.
In one implementation, the log processing device may also send the unresolved log files to an intermediary device (e.g., a high-throughput distributed publish-subscribe message system (Apache Kafka)), which may be used to quickly process messages between the log processing device and the management device to improve processing efficiency of log information.
It should be noted that the sending condition includes that the current time is within a preset sending time period, and the number of sent pieces within the preset sending time period is smaller than a preset threshold of the number of sent pieces. For example, the following fields are employed as configuration information of the transmission condition: < samples-kafka enable ═ true "num ═ 10" interval ═ 1"unit ═ min" topoic ═ sdap-samples-unit "/>.
Wherein kafka is a high-throughput distributed publish-subscribe messaging system for processing messages between log processing means and management means at high speed. Enable indicates whether to send to kafka; num represents a preset threshold of the number of sending strips; interval represents a transmission time interval; unit represents a unit of time interval (e.g., 1 minute interval, min for minutes, sec for seconds); topic indicates the topic sent to kafka, i.e. what the sending condition is for, e.g. if topic is sdap-samples-match, it indicates that the topic currently sent to kafka is a sample mismatch of the service discovery application profile. Each time an unresolved log file is successfully sent to kafka, the number of sent pieces needs to be automatically increased by one; and in the preset sending time period, if the number of the sending pieces is equal to the preset threshold of the number of the sending pieces, the unresolved log file is not sent to the kafka any more, and the next sending time interval is waited for. If it is determined that the unresolved log file satisfies the sending condition, step 130 is executed, otherwise, step 140 is executed.
And step 130, generating and sending a log reporting message to the management device according to the unresolved log file and the attribute information of the terminal equipment.
It should be noted that, after receiving the log reporting message, the management device parses the log reporting message to obtain the unanalyzed log files and the attribute information of the terminal device, then establishes a storage folder according to the attribute information and the information such as the IP address of the log processing device, stores these unanalyzed log files, and then processes the unanalyzed log files according to the log management policy.
Step 140, storing the unanalyzed log file to the local, and processing the unanalyzed log file according to the storage time of the unanalyzed log file.
The log processing device may sort the unanalyzed log files according to the storage time of the unanalyzed log files, and delete the unanalyzed log file with the earliest storage time when a certain condition is satisfied. For example, three unresolved log files are saved, and the storage time of the three unresolved log files is, in order, 10 months, 2 days, 15: 30. 10 month, 2 days 17: 30 and 10 months, 2 days 20: 30, if the unanalyzed log file needs to be deleted currently, the storage time of the unanalyzed log file deleted first is 10 months, 2 days, 15: 30 to the unresolved log file.
In some implementations, the stored unresolved log file is stored in the log processing device in a file directory manner, and step 140 may be implemented by the following steps, including steps 141-142.
Step 141, saving the unanalyzed log file to a local first preset folder.
The first preset folder is a folder established according to the attribute information of the terminal device. For example, the attribute information of the terminal device includes the type of the terminal device (expressed using sensorModel); and establishing a first preset folder according to the type of the terminal equipment, and storing the unresolved log file into the first preset folder, wherein the storage path of the first preset folder is/sensorModel.
And 142, deleting the unanalyzed log file at intervals of a first preset time according to the storage time of the unanalyzed log file, the storage capacity of the first preset folder and the size of the unanalyzed log file.
It should be noted that the first preset time period may be set to different time periods of 1 hour, 5 hours, 1 day, or 2 days. If the log information reported by the terminal device is frequent and more unanalyzed log files are obtained, the first preset time can be properly set to be shorter time, such as 1 hour and the like; if the log information reported by the terminal device is not much and the number of the obtained unanalyzed log files is not much, the first preset time duration can be properly set to be a longer time duration, for example, 5 days. The first preset time length is only illustrated, and may be specifically set according to an actual situation, and other first preset time lengths not illustrated are also within the protection scope of the present application, and are not described herein again.
In some embodiments, the size of each unresolved log file is the same, and the number of unresolved log files stored in the first preset folder can be calculated and obtained through the storage capacity of the first preset folder and the size of each unresolved log file.
In some embodiments, the storage capacity of the first preset folder is monitored; comparing the storage capacity of the first preset folder with a first storage threshold value; if the storage capacity of the first preset folder is determined to be larger than a first storage threshold, calculating the number of first-class files to be deleted according to the size of the unresolved log files, the first storage threshold and the storage capacity of the first preset folder, wherein the number of the first-class files to be deleted is the calculated number of the unresolved log files to be deleted; sequencing the unresolved log files stored in a first preset folder according to the storage time of the unresolved log files to obtain a first sequencing result; and deleting the unresolved log files according to the number of the first type of files to be deleted and the first sequencing result.
For example, a timing task is started, the storage capacity of a first preset folder is checked regularly, when the storage capacity of the first preset folder is greater than a first storage threshold, all files in the first preset folder are sorted according to storage time, from the file with the earliest storage time, an unresolved log file is deleted, and the number of the unresolved log files to be deleted is calculated by adopting the following formula: the number of the unresolved log files to be deleted is: the number of the unresolved log files to be deleted is (the storage capacity of the first preset folder-the first storage threshold/2)/the capacity of each unresolved log file.
In this embodiment, by determining whether the obtained unanalyzed log file meets the sending condition, if yes, generating and sending a log reporting message to the management device according to the unanalyzed log file and the attribute information of the terminal device corresponding to the unanalyzed log file, so that the management device processes the unanalyzed log file according to a log management policy; the method has the advantages that the unanalyzed log files meeting the sending conditions can be sent to the management device for processing, the processing pressure of the log processing device is reduced, and the processing efficiency of the accumulated log information is improved; if the unanalyzed log file does not meet the sending condition, the unanalyzed log file is stored to the local, and the unanalyzed log file is processed according to the storage time of the unanalyzed log file, so that the residual unanalyzed log file in the accumulated log information can be processed in time, the condition that the storage space is occupied is reduced, and the efficiency of log analysis is improved.
Example two
The embodiment of the present application provides a log processing method, which is substantially the same as the first embodiment, and the main difference is that: the log processing device needs to acquire a processing rule issued by the management device and process the log information according to the processing rule.
Fig. 2 is a flowchart of a log processing method in this embodiment, which can be applied to a log processing apparatus. The method may specifically comprise the following steps.
Step 210, in response to the configuration message sent by the management device, a processing rule is obtained.
The processing rule is a rule verified by the management apparatus. When the management device inquires the internally stored processing rules according to the address information and the identification of the terminal equipment, and when the latest rule file of the log information corresponding to the terminal equipment conforms to a rule writing specification (for example, a writing specification in an xml format) and contains the latest processing rule of the log information corresponding to the terminal equipment, the verification is passed; and generating a configuration message according to the processing rule passing the verification, and sending the configuration message to the log processing device. The log processing device analyzes the received rule file to obtain a processing rule passing the verification.
Step 220, obtaining the unanalyzed log file and the attribute information of the terminal device corresponding to the unanalyzed log file.
It should be noted that the unresolved log file includes two types: a completely unresolved log file and a partially unresolved log file; the log file which is not analyzed completely is the log file which can not be analyzed by using a preset processing rule after the log processing device collects the log information; the partially unresolved log file is a file in which after the log processing device collects log information, a part of the log information can be analyzed by using a processing rule configured in advance, but a part of the log information cannot be analyzed by using the processing rule.
In some implementations, obtaining an unresolved log file includes: collecting log information reported by terminal equipment; judging whether the log information can be processed by using the processing rule; if it is determined that the log information cannot be processed using the processing rule, the log information is saved to an unresolved log file.
The log information is data of a character string type, and when the log information cannot be processed by using a pre-stored processing rule (for example, when the log information cannot be analyzed by using the processing rule), the log information can be written into an unresolved log file, so that the purpose of saving the log information is achieved, and the log information can be conveniently processed subsequently.
Step 230, determine whether the unanalyzed log file meets the sending condition.
If it is determined that the unresolved log file satisfies the sending condition, step 240 is executed, otherwise, step 250 is executed.
And step 240, generating and sending a log reporting message to the management device according to the unresolved log file and the attribute information of the terminal equipment.
Step 250, storing the unanalyzed log file to the local, and processing the unanalyzed log file according to the storage time of the unanalyzed log file.
It should be noted that, the contents of the steps 230 to 250 in this embodiment are the same as the contents of the steps 120 to 140 in the first embodiment, and are not described herein again.
In this embodiment, a processing rule is obtained by receiving a configuration message sent by a management device, whether the collected log information matches the processing rule is judged, and when the collected log information does not match the processing rule, an unanalyzed log file is obtained, so that the unanalyzed log file can be more accurately obtained, the processing efficiency of the log information is improved, the occupied storage space of the log processing device is reduced, and the log analysis efficiency is improved.
EXAMPLE III
An embodiment of the present application provides a log processing method, and fig. 3 is a flowchart of the log processing method in the embodiment, where the method is applicable to a management device. The method may specifically comprise the following steps.
Step 310, responding to the log reporting message sent by the log processing device, and obtaining an unresolved log file.
The log reporting information includes an unresolved log file, address information of a log processing device, address information of a terminal device corresponding to the unresolved log file, a device type of the terminal device, and the like. The unanalyzed log files can be stored in a classified mode according to the address information of the log processing device, the address information of the terminal equipment and the equipment type of the terminal equipment as storage labels, and the subsequent processing of the unanalyzed log files is facilitated.
And step 320, processing the unanalyzed log file according to the log management strategy.
The log management policy may be to delete the accumulated unanalyzed log files at regular time, or may be to process the unanalyzed log files by other policies set manually. The above are only examples, and other unexplained log management policies are also within the protection scope of the present application, and may be specifically set according to actual situations, and are not described herein again.
In some embodiments, step 320 may be implemented by the following steps, specifically including step 321 and step 322.
Step 321, establishing a second preset folder according to the address information of the log processing device; and storing the unresolved log file into a second preset folder.
For example, after acquiring the unresolved log file, the management device stores the unresolved log file in a second preset folder, where the storage path of the second preset folder may be platformID/collectionip/sensorIP/sensorModel, where platformID represents an ID of the management device, collectionip represents an IP of the log processing device, sensorIP represents an IP of a terminal device accessing the log processing device, and sensorModel represents a type of the terminal device.
And 322, deleting the unanalyzed log file at intervals of a second preset time according to the storage capacity of the second preset folder and the log management strategy.
Specifically, the second preset time period may be a time period of 5 hours, 10 hours, and the like, for example, every 10 hours, the unanalyzed log file stored first in the second preset folder is deleted, so as to prevent accumulation of the unanalyzed log files and improve the use efficiency of the storage space.
In some embodiments, the storage capacity of the second preset folder is monitored; comparing the storage capacity of the second preset folder with a second storage threshold; if the storage capacity of the second preset folder is determined to be larger than a second storage threshold, calculating the number of second files to be deleted according to the size of the unresolved log files, the second storage threshold and the storage capacity of the second preset folder, wherein the number of the second files to be deleted is the number of the unresolved log files to be deleted; sequencing the unanalyzed log files stored in the second preset folder according to the storage time of the unanalyzed log files to obtain a second sequencing result; and deleting the unresolved log files according to the second sequencing result and the number of the second type files needing to be deleted.
For example, the log processing apparatus writes the log information of the terminal device into an unresolved log file of the following path of the log processing apparatus: collectotropip/sensorIP/sensorMode/FaildMatched _ x.log. The unresolved log file may be named FaildMathed _ x.log, where x starts at 0 and is a positive integer. The following configuration information may be used to configure the preset file capacity threshold of the unanalyzed log file and the total capacity threshold of the second preset folder: < samples-savesize ═ 10M "totalsize ═ 500G"/>. Wherein, the size represents a preset file capacity threshold of the unresolved log file; totosilze represents a total capacity threshold for the second preset folder.
Before writing a file every time, whether the file exists or not needs to be judged, if the file FaildMathed _ x.log exists, log information is directly written, and if the file FaildMathed _ x.log does not exist, a directory and a file need to be created first, and then the log information is written. Then judging the capacity of the current file, and when the capacity of the file is smaller than a preset file capacity threshold value, directly writing the file; when the capacity size of the file exceeds a preset file capacity threshold value, a log file needs to be created again, and the value of x is added with 1, namely log files FaildMatched _0.log, FaildMatched _1.log, … … and FaildMatched _ x.log are sequentially created. The storage mode can store the unresolved log information reported by each log processing device in a classified manner, so that the management of the log information is facilitated, and the storage confusion of the log information is avoided.
In the embodiment, the unanalyzed log files are obtained by receiving the log reporting messages sent by the log processing devices, and then the unanalyzed log files reported by the log processing devices are classified and stored by adopting a classified storage mode, so that the storage confusion of log information is avoided, and the log files are convenient to search and manage; when the storage capacity of a second preset folder for storing the unanalyzed log files is monitored to be larger than a first storage threshold value, the number of second files to be deleted is calculated by using a calculation formula, and then the unanalyzed log files stored in the second preset folder at first are deleted according to the number, so that the accumulation of the unanalyzed log files is prevented, and the use efficiency of a storage space is improved.
Example four
The embodiment of the present application provides a log processing method, which is substantially the same as the third embodiment, and the main difference is that: before obtaining the unresolved log file, a processing rule is required to be obtained, and whether the processing rule meets the rule writing specification is judged; when the processing rule meets the rule writing specification, the processing rule is sent to the log processing device, so that the log processing device can determine the unanalyzed log file according to the processing rule, and the processing efficiency of log information is improved.
Fig. 4 is a flowchart of a log processing method in this embodiment, which can be applied to a management device. The method may specifically comprise the following steps.
At step 410, processing rules are obtained.
The processing rule is a rule reported to the management device by the log processing device when the terminal device initially accesses the log processing device.
Step 420, determine whether the processing rule meets the rule writing specification.
The processing rule acquired in step 410 does not necessarily conform to the rule writing specification, and therefore it is necessary to check whether the processing rule conforms to the rule writing specification, for example, whether the processing rule conforms to the writing specification of the xml format, whether the processing rule corresponds to log information that the log processing apparatus desires to parse, and the like.
And step 430, if the processing rule is determined to be in accordance with the rule writing specification, generating and sending a configuration message to the log processing device according to the processing rule.
Specifically, when the rule file of the latest log information corresponding to the terminal device that is queried conforms to the rule writing specification and contains the processing rule of the latest log information corresponding to the terminal device, it indicates that the processing rule conforms to the rule writing specification. When the log processing device receives the processing rule issued by the management device, the log processing device processes the log information reported by the terminal equipment according to the processing rule to obtain an unresolved log file; if the log processing device cannot process the unresolved log file in time according to the processing capacity of the log processing device, a log reporting message is generated and sent to the management device according to the unresolved log file, and meanwhile, the unresolved log file which is locally stored and sent to the management device is deleted, so that the storage space is saved, and the processing performance of the log processing device is improved.
Step 440, responding to the log reporting message sent by the log processing device, obtaining the unanalyzed log file.
It should be noted that step 440 in this embodiment is the same as step 310 in the third embodiment, and is not described herein again.
Step 450, processing the unanalyzed log file according to the log management strategy.
It should be noted that, when the processing performance of the management device reaches the limit, the unanalyzed log file corresponding to the log processing device is deleted according to the sequence of the storage time, so as to achieve the purpose of releasing the storage space, thereby improving the processing performance of the management device.
In this embodiment, the latest processing rule is determined by judging whether the acquired latest processing rule meets the rule writing specification, and a configuration message is generated and sent to the log processing device according to the processing rule, so that the log processing device can process the log information reported by each terminal device according to the latest processing rule, and the processing speed of the log information is increased; when the number of the unanalyzed log files stored by the management device is too large, the unanalyzed log files corresponding to the log processing device can be deleted according to the sequence of the storage time of the unanalyzed log files so as to achieve the purpose of releasing the storage space, thereby improving the processing performance of the management device.
EXAMPLE five
Fig. 5 is a schematic structural diagram of a log processing apparatus according to an embodiment of the present application, and for specific implementation of the apparatus, reference may be made to the related description of the first embodiment or the second embodiment, and repeated descriptions are omitted. It should be noted that the specific implementation of the apparatus in this embodiment is not limited to the above embodiment, and other undescribed embodiments are also within the scope of the apparatus.
As shown in fig. 5, the log processing apparatus specifically includes: the first obtaining module 501 is configured to obtain an unanalyzed log file and attribute information of a terminal device corresponding to the unanalyzed log file; the judging module 502 is configured to judge whether the unanalyzed log file meets the sending condition, and obtain a judgment result; the sending module 503 is configured to generate and send a log reporting message to the management device according to the unresolved log file and the attribute information of the terminal device when the unresolved log file meets the sending condition as a result of the determination, so that the management device processes the unresolved log file according to a log management policy; the first processing module 504 is configured to, when the determination result is that the unanalyzed log file does not satisfy the sending condition, store the unanalyzed log file to the local, and process the unanalyzed log file according to the storage time of the unanalyzed log file.
In the embodiment, whether the obtained unanalyzed log file meets the sending condition is judged by the judging module, if so, the sending module is used for generating and sending a log reporting message to the management device according to the unanalyzed log file and the attribute information of the terminal equipment corresponding to the unanalyzed log file, so that the management device processes the unanalyzed log file according to a log management strategy; the method has the advantages that the unanalyzed log files meeting the sending conditions can be sent to the management device for processing, the processing pressure of the log processing device is reduced, and the processing efficiency of the accumulated log information is improved; if the unanalyzed log file does not meet the sending condition, the unanalyzed log file is stored to the local by using the first processing module, and the unanalyzed log file is processed according to the storage time of the unanalyzed log file, so that the residual unanalyzed log file in the accumulated log information can be processed in time, the condition that the storage space is occupied is reduced, and the efficiency of log analysis is improved.
It should be understood that this embodiment is an apparatus embodiment corresponding to the first embodiment or the second embodiment, and may be implemented in cooperation with the first embodiment or the second embodiment. Related technical details mentioned in the first embodiment or the second embodiment are still valid in this embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related art details mentioned in the present embodiment can also be applied to the first embodiment or the second embodiment.
EXAMPLE six
Fig. 6 is a schematic structural diagram of a management device according to an embodiment of the present application, and for specific implementation of the management device, reference may be made to related descriptions of the third embodiment or the fourth embodiment, and repeated descriptions are omitted here. It should be noted that the specific implementation of the apparatus in this embodiment is not limited to the above embodiment, and other undescribed embodiments are also within the scope of the apparatus.
As shown in fig. 6, the management apparatus specifically includes: the second obtaining module 601 is configured to obtain an unanalyzed log file in response to a log reporting message sent by the log processing apparatus; the second processing module 602 is configured to process the unanalyzed log file according to a log management policy.
In the embodiment, the obtained unanalyzed log file is processed through the log management strategy, so that the processing speed of the unanalyzed log file is increased, the accumulation of log information is avoided, the condition that the storage space of the management device is occupied is further reduced, and the system performance of the management device is improved.
It should be understood that this embodiment is an apparatus example corresponding to the third embodiment or the fourth embodiment, and may be implemented in cooperation with the third embodiment or the fourth embodiment. Related technical details mentioned in the third embodiment or the fourth embodiment are still valid in the present embodiment, and are not described herein again in order to reduce repetition. Accordingly, the details of the related art mentioned in this embodiment can also be applied to the third embodiment or the fourth embodiment.
It should be noted that each module referred to in this embodiment is a logical module, and in practical applications, one logical unit may be one physical unit, may be a part of one physical unit, and may be implemented by a combination of multiple physical units. In addition, in order to highlight the innovative part of the present application, a unit that is not so closely related to solving the technical problem proposed by the present application is not introduced in the present embodiment, but it does not indicate that no other unit exists in the present embodiment.
EXAMPLE seven
An embodiment of the present application provides a log processing system, as shown in fig. 7, including: terminal device a711, terminal device B712, log collector a721, log collector B722, and management platform device 730. The terminal device a711 and the terminal device B712 report their own log information to the log collector a721, and the log collector a721 obtains the processing rule from the management platform device 730, and determines whether the processing rule can resolve unmatched logs, to obtain a determination result; further, the log collector a721 processes the log information according to the judgment result to obtain an unanalyzed log file; the unanalyzed log files meeting the sending conditions are sent to the management platform device 730 for processing, and the unanalyzed log files not meeting the sending conditions are stored locally for processing.
It should be noted that one management platform device 730 may be connected to a plurality of log collectors (e.g., log collector a721 and log collector B722), and each log collector may be connected to and obtain log information of a plurality of terminal devices (e.g., terminal device a711 and terminal device B712, etc.).
Fig. 8 is a flowchart of a working method of the log processing system in this embodiment, which may specifically include the following steps.
In step 801, terminal device a711 initially accesses log collector a 721.
In step 802, terminal device B712 initially accesses log collector a 721.
In step 803, the log collector a721 generates a rule report message corresponding to the terminal device a711 and a rule report message corresponding to the terminal device a712 according to the rule files of the terminal device a711 and the terminal device B712, the management platform ID, the IP of the log collector, the IP of the terminal device, and the like, and sends the two rule report messages to the management platform device 730.
It should be noted that the rule reporting message corresponding to the terminal device a711 includes a rule file for processing log information of the terminal device a711 with the device type a, and the rule reporting message corresponding to the terminal device a712 includes a rule file for processing log information of the terminal device B712 with the device type B. If there are other terminal devices accessing the log collector a721, after the other terminal devices initially access, the log collector a721 reports the rule file corresponding to the terminal device to the management platform device 730. I.e. each rule file corresponds to only one device type of terminal device accessing one collector. And each rule file is put into the rule reporting message in the form of a compressed packet.
Step 804, after receiving the rule reporting message, the management platform device 730 parses the rule reporting message to obtain information of a first rule file, a second rule file management platform ID, an IP of the log collector, an IP of the terminal device, and the like, and establishes a second preset folder according to the management platform ID, the IP of the log collector, and the IP of the terminal device, and then decompresses the first rule file and the second rule file, and stores the decompressed first rule file and first rule file in the corresponding second preset folder, respectively.
As shown in table 1, the attribute information of each terminal device includes: platformID, collectrIP, sensorIP and sensorModel, wherein the platformID represents the ID of the management platform device 730, the collectrIP represents the IP of the log collector A721 or the log collector A722, the sensorIP represents the IP of the terminal device A711 or the terminal device B712, and the sensorModel represents the type of the terminal device A711 or the terminal device B712.
TABLE 1 Attribute information
Figure BDA0002419345180000171
Figure BDA0002419345180000181
For example, if the platformID of the management platform device 730 is 101, the collectionerip of the log collector a721 is 192.168.1.5, the sensorIp of the terminal device a711 is 192.168.2.8, and the sensorModel of the terminal device a711 is a, the corresponding first rule file of the terminal device a711 will be stored in the file path: 101/192.168.1.5/192.168.2.8/A/; similarly, if the sensorIP of terminal B712 is 192.168.2.9 and the sensorModel of terminal B712 is B, the corresponding second rule file of terminal B712 would be stored in the file path: 101/192.168.1.5/192.168.2.9/B/.
It should be noted that, if a file path exists, the rule file may be directly stored in the corresponding file path, and if the file path does not exist, a directory and a corresponding folder need to be created according to the attribute information, and then the rule file is stored in the corresponding folder.
In step 805, the terminal apparatus a711 transmits log information to the log collector a 721.
In step 806, when determining that the processing rule of the log information corresponding to the terminal device a711 is updated, the management platform device 730 actively issues a configuration message to the log collector a 721.
The configuration message includes, among other things, the latest processing rule of the log information corresponding to the terminal device a711, the identification and address information of the terminal device a711, and the like.
It should be noted that before the management platform device 730 sends the configuration message to the log collector a721, the processing rule therein needs to be verified, and when the rule file of the latest log information corresponding to the terminal device a711, which is queried, conforms to the rule writing specification (for example, the writing specification in xml format), and contains the latest processing rule of the log information corresponding to the terminal device a711, it indicates that the verification is passed, and the rule file that is verified can be sent to the log collector a 721.
In step 807, the log collector a721 processes the received log information sent by the terminal device a711 according to the processing rule issued by the management platform device 730.
It should be noted that, when receiving the configuration message sent by the management platform device 730, the log collector a721 needs to first determine whether the identifier of the management platform device carried in the configuration message is the identifier of the management platform device 730 and whether the IP address of the log collector is the IP address of the log collector a721, only when the identifier of the management platform device carried in the configuration message is the identifier of the management platform device 730 and the IP address of the log collector is the IP address of the log collector a721, it can be determined that the configuration message is sent to the log collector a721 by the management platform device 730, and when the log collector a721 determines that the processing rule carried in the configuration message is to be applied to process the unresolved log file, the globally unique processing rule of the log collector a721 is updated to facilitate the processing of the unresolved log file, and after updating the processing rule, the log collector a721 does not need to be restarted, improving the processing efficiency of the log collector a 721.
Specifically, the log information is data of a character string type, and may be defined as original. Firstly, saving the log information as an unanalyzed log file, establishing a first preset folder according to the attribute information of the terminal equipment corresponding to the log information, and writing the log information into the corresponding log file in the first preset folder. The log collector a721 writes the log information of the terminal device a711 into an unresolved log file of the following path of the log collector a 721: A711/FaildMatched _ x.log. The unresolved log file may be named FaildMathed _ x.log, where x starts at 0 and is a positive integer. The following configuration information may be used to configure the preset file capacity threshold of the unresolved log file and the total capacity threshold of the first preset folder: < samples-savesize ═ 10M "totalsize ═ 500G"/>. Wherein, the size represents a preset file capacity threshold of the unresolved log file; totolsize represents the total capacity threshold of the first preset folder.
Before writing a file every time, whether the file exists or not needs to be judged, if the file FaildMathed _ x.log exists, log information is directly written, and if the file FaildMathed _ x.log does not exist, a directory and a file need to be created first, and then the log information is written. Then judging the capacity of the current file, and when the capacity of the file is smaller than a preset file capacity threshold value, directly writing the file; when the capacity size of the file exceeds a preset file capacity threshold value, a log file needs to be created again, and the value of x is added with 1, namely log files FaildMatched _0.log, FaildMatched _1.log, … … and FaildMatched _ x.log are sequentially created. The storage mode can store the unresolved log information reported by each log collector in a classified manner under the condition of multiple platforms and multiple log collectors, so that the storage confusion of the log information is avoided, and the management of the log information is facilitated.
Then, judging whether the initial processing rule is matched with the unanalyzed log file, if so, analyzing the unanalyzed log file by using the initial processing rule, and extracting information required by the log collector a 721; if not, then judging whether the unanalyzed log file meets the sending condition, and if the unanalyzed log file does not meet the sending condition, executing step 808; when the unresolved log file satisfies the sending condition, step 809 is performed.
In step 808, the log collector A721 processes the unresolved log file according to the storage time of the unresolved log file.
Specifically, the storage capacity of a first preset folder is monitored; comparing the storage capacity of the first preset folder with totalsize; if the storage capacity of the first preset folder is determined to be larger than totalsize, calculating and obtaining the number of the unresolved log files needing to be deleted according to the size of the unresolved log files, totalsize and the storage capacity of the first preset folder; sequencing the unresolved log files stored in a first preset folder according to the storage time of the unresolved log files to obtain a first sequencing result; and deleting the unresolved log files according to the number of the unresolved log files needing to be deleted and the first sequencing result.
For example, a timing task is started, the storage capacity of a first preset folder is checked regularly, when the storage capacity of the first preset folder is greater than totalsize, all files in the first preset folder are sorted according to storage time, an unresolved log file is deleted from the file with the earliest storage time, and the number of the unresolved log files to be deleted is calculated by adopting the following formula: the number of the unresolved log files to be deleted is: the number of the unresolved log files to be deleted is (storage capacity-totalsize/2 of the first preset folder)/the capacity of each unresolved log file.
In step 809, the log collector a721 generates and sends a log report message to the management platform device 730 according to the unanalyzed log file and the attribute information of the terminal device a 711.
In step 810, after receiving the log report message, the management platform device 730 processes the unanalyzed log file according to the log management policy.
Specifically, a second preset folder is established according to the address information of the log collector a 721; and storing the unresolved log file into a second preset folder. It should be noted that the storing method is the same as the storing method in step 807, and all the storing method is to store the data in the folder corresponding to the address information of the log collector a721, for example, the path of the second preset folder is platformID/collectitorip/sensorIP/sensorModel/.
Monitoring the storage capacity of a second preset folder every second preset time interval; comparing the storage capacity of the second preset folder with a second storage threshold; if the storage capacity of the second preset folder is determined to be larger than a second storage threshold, calculating the number of second files to be deleted according to the size of the unresolved log files, the second storage threshold and the storage capacity of the second preset folder, wherein the number of the second files to be deleted is the number of the unresolved log files to be deleted; sequencing the unanalyzed log files stored in the second preset folder according to the storage time of the unanalyzed log files to obtain a second sequencing result; and deleting the unresolved log files according to the second sequencing result and the number of the second type files needing to be deleted.
The number of the second files to be deleted is calculated by adopting the following formula: the number of the second type files to be deleted is (the storage capacity of the second preset folder-the second storage threshold/2)/the capacity of each unresolved log file.
In this embodiment, the log collector processes the log information reported by the terminal device by obtaining the latest processing rule issued by the management platform device, and obtains and stores the unanalyzed log files, and then when the number of the unanalyzed log files stored is too large, a certain number of unanalyzed log files are deleted according to the storage time of the unanalyzed log files and the storage capacity of the first preset folder, so that the storage space of the log collector is released, the accumulation of log information is avoided, the processing pressure of the log processing device is reduced, and the processing efficiency of the accumulated log information is improved. Meanwhile, when receiving the unanalyzed log files reported by the log collector, the management platform equipment deletes a certain number of unanalyzed log files according to the log management strategy, releases the storage space of the management platform equipment and improves the performance of the management platform equipment.
Example eight
The embodiment of the application provides electronic equipment. Fig. 9 is a block diagram of an exemplary hardware architecture of an electronic device that can implement a log processing method according to an embodiment of the present application.
As shown in fig. 9, electronic device 900 includes an input device 901, an input interface 902, a central processor 903, a memory 904, an output interface 905, and an output device 906. The input interface 902, the central processing unit 903, the memory 904, and the output interface 905 are connected to each other through a bus 907, and the input device 901 and the output device 906 are connected to the bus 907 through the input interface 902 and the output interface 905, respectively, and further connected to other components of the electronic device 900.
Specifically, the input device 901 receives input information from the outside (e.g., a log processing apparatus or a management apparatus) and transmits the input information to the central processor 903 through the input interface 902; central processor 903 processes input information based on computer-executable instructions stored in memory 904 to generate output information, stores the output information temporarily or permanently in memory 904, and then transmits the output information to output device 906 via output interface 905; output device 906 outputs the output information external to computing device 900 for use by a user.
In one embodiment, the electronic device 900 shown in fig. 9 may be implemented as a network device that may include: a memory configured to store a program; a processor configured to execute the program stored in the memory to perform any one of the log processing methods described in the above embodiments.
According to an embodiment of the application, the process described above with reference to the flow chart may be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network, and/or installed from a removable storage medium.
It will be understood by those of ordinary skill in the art that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof. In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be performed by several physical components in cooperation. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). The term computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, as is well known to those of ordinary skill in the art. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media as known to those skilled in the art.
It is to be understood that the above embodiments are merely exemplary embodiments that are employed to illustrate the principles of the present application, and that the present application is not limited thereto. It will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the application, and these changes and modifications are to be considered as the scope of the application.

Claims (16)

1. A method of log processing, the method comprising:
acquiring an unresolved log file and attribute information of terminal equipment corresponding to the unresolved log file;
judging whether the unresolved log file meets a sending condition;
if so, generating and sending a log reporting message to a management device according to the unresolved log file and the attribute information of the terminal equipment, so that the management device processes the unresolved log file according to a log management strategy;
otherwise, the unanalyzed log file is saved to the local, and the unanalyzed log file is processed according to the storage time of the unanalyzed log file.
2. The method of claim 1, wherein saving the unresolved log file to a local location and processing the unresolved log file according to a storage time of the unresolved log file comprises:
storing the unanalyzed log file into a local first preset folder, wherein the first preset folder is a folder established according to the attribute information of the terminal equipment;
and deleting the unresolved log file according to the storage time of the unresolved log file, the storage capacity of the first preset folder and the size of the unresolved log file every a first preset time interval.
3. The method of claim 2, wherein deleting the unresolved log file according to a storage time of the unresolved log file, a storage capacity of the first predetermined folder, and a size of the unresolved log file comprises:
monitoring the storage capacity of the first preset folder;
comparing the storage capacity of the first preset folder with a first storage threshold value;
if the storage capacity of the first preset folder is determined to be larger than the first storage threshold, calculating the number of first-class files to be deleted according to the size of the unresolved log files, the first storage threshold and the storage capacity of the first preset folder, wherein the number of the first-class files to be deleted is the calculated number of the unresolved log files to be deleted;
sequencing the unresolved log files stored in the first preset folder according to the storage time of the unresolved log files to obtain a first sequencing result;
and deleting the unresolved log files according to the number of the first type files to be deleted and the first sequencing result.
4. The method according to claim 1, wherein before the step of obtaining the unresolved log file and the attribute information of the terminal device corresponding to the unresolved log file, the method further comprises:
obtaining a processing rule in response to a configuration message sent by the management device, wherein the processing rule is a rule verified by the management device.
5. The method of claim 4, wherein obtaining the unresolved log file comprises:
collecting log information reported by the terminal equipment;
judging whether the log information can be processed by using the processing rule;
and if the log information cannot be processed by using the processing rule, saving the log information into the unresolved log file.
6. The method according to any of claims 1 to 5, wherein the transmission condition comprises: the current moment is in a preset sending time period, and the number of sent pieces in the preset sending time period is smaller than a preset threshold of the number of sent pieces.
7. The method according to any one of claims 1 to 5, wherein the attribute information of the terminal device at least comprises any one of an identifier of the terminal device, type information of the terminal device, and address information of the terminal device.
8. A method of log processing, the method comprising:
responding to a log reporting message sent by a log processing device, and acquiring an unresolved log file;
and processing the unresolved log file according to a log management strategy.
9. The method of claim 8, wherein processing the unresolved log file in accordance with a log management policy comprises:
establishing a second preset folder according to the address information of the log processing device;
storing the unanalyzed log file into the second preset folder;
and deleting the unanalyzed log file at every interval of a second preset time according to the storage capacity of the second preset folder and the log management strategy.
10. The method of claim 9, wherein deleting the unresolved log file according to the storage capacity of the second predetermined folder and the log management policy comprises:
monitoring the storage capacity of the second preset folder;
comparing the storage capacity of the second preset folder with a second storage threshold;
if the storage capacity of the second preset folder is determined to be larger than the second storage threshold, calculating the number of second files to be deleted according to the size of the unresolved log file, the second storage threshold and the storage capacity of the second preset folder, wherein the number of the second files to be deleted is the calculated number of the unresolved log files to be deleted;
sequencing the unanalyzed log files stored in the second preset folder according to the storage time of the unanalyzed log files to obtain a second sequencing result;
and deleting the unresolved log file according to the second sorting result and the number of the second files to be deleted.
11. The method of claim 8, wherein before the step of obtaining the unresolved log file in response to the log reporting message sent by the log processing device, the method further comprises:
acquiring a processing rule;
judging whether the processing rule conforms to a rule writing specification;
if the processing rule is determined to meet the rule writing specification, generating and sending a configuration message to the log processing device according to the processing rule so that the log processing device can obtain the processing rule, and processing the log information reported by the terminal equipment according to the processing rule to obtain the unresolved log file.
12. The method of claim 11, wherein processing the unresolved log file in accordance with a log management policy further comprises:
and deleting the unresolved log file corresponding to the log processing device.
13. A log processing apparatus, comprising:
the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is used for acquiring an unresolved log file and attribute information of terminal equipment corresponding to the unresolved log file;
the judging module is used for judging whether the unresolved log file meets the sending condition or not and obtaining a judging result;
a sending module, configured to generate and send a log report message to a management device according to the unresolved log file and the attribute information of the terminal device when the determination result indicates that the unresolved log file satisfies a sending condition, so that the management device processes the unresolved log file according to a log management policy;
and the first processing module is used for saving the unanalyzed log file to the local and processing the unanalyzed log file according to the storage time of the unanalyzed log file when the judgment result shows that the unanalyzed log file does not meet the sending condition.
14. A management device, comprising:
the second acquisition module is used for responding to the log reporting message sent by the log processing device and acquiring the unresolved log file;
and the second processing module is used for processing the unresolved log file according to the log management strategy.
15. An electronic device, comprising:
one or more processors;
storage means having one or more programs stored thereon which, when executed by the one or more processors, cause the one or more processors to carry out a method according to any one of claims 1 to 7, or a method according to any one of claims 8 to 12.
16. A computer-readable medium, on which a computer program is stored which, when executed by a processor, implements the method of any one of claims 1 to 7, or the method of any one of claims 8 to 12.
CN202010200916.7A 2020-03-20 2020-03-20 Log processing method and device, electronic equipment and readable medium Pending CN111309696A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010200916.7A CN111309696A (en) 2020-03-20 2020-03-20 Log processing method and device, electronic equipment and readable medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010200916.7A CN111309696A (en) 2020-03-20 2020-03-20 Log processing method and device, electronic equipment and readable medium

Publications (1)

Publication Number Publication Date
CN111309696A true CN111309696A (en) 2020-06-19

Family

ID=71157430

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010200916.7A Pending CN111309696A (en) 2020-03-20 2020-03-20 Log processing method and device, electronic equipment and readable medium

Country Status (1)

Country Link
CN (1) CN111309696A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111913885A (en) * 2020-08-07 2020-11-10 腾讯科技（深圳）有限公司 Log processing method and device, computer readable storage medium and equipment
CN113015203A (en) * 2021-03-22 2021-06-22 Oppo广东移动通信有限公司 Information acquisition method, device, terminal, system and storage medium
CN114528258A (en) * 2022-02-18 2022-05-24 北京百度网讯科技有限公司 Asynchronous file processing method, device, server, medium, product and system

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080086561A1 (en) * 2005-04-27 2008-04-10 Huawei Technologies Co., Ltd. Method for obtaining log information from network element device by network management server, a network element device and a network management server
CN102799514A (en) * 2011-05-24 2012-11-28 中兴通讯股份有限公司 Method and system for managing log records
CN103678570A (en) * 2013-12-10 2014-03-26 中国人民解放军理工大学 Multi-level storage and recovery method and system of journal file in cloud environment
WO2018006721A1 (en) * 2016-07-08 2018-01-11 中兴通讯股份有限公司 Method and apparatus for storing log file
CN108170584A (en) * 2017-12-26 2018-06-15 广东欧珀移动通信有限公司 log processing method, device, storage medium and terminal device
CN110019239A (en) * 2017-12-29 2019-07-16 百度在线网络技术（北京）有限公司 Storage method, device, electronic equipment and the storage medium of reported data
CN110555005A (en) * 2019-08-12 2019-12-10 中国南方电网有限责任公司 Extensible open type paradigm method and device applied to power grid monitoring system

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080086561A1 (en) * 2005-04-27 2008-04-10 Huawei Technologies Co., Ltd. Method for obtaining log information from network element device by network management server, a network element device and a network management server
CN102799514A (en) * 2011-05-24 2012-11-28 中兴通讯股份有限公司 Method and system for managing log records
CN103678570A (en) * 2013-12-10 2014-03-26 中国人民解放军理工大学 Multi-level storage and recovery method and system of journal file in cloud environment
WO2018006721A1 (en) * 2016-07-08 2018-01-11 中兴通讯股份有限公司 Method and apparatus for storing log file
CN108170584A (en) * 2017-12-26 2018-06-15 广东欧珀移动通信有限公司 log processing method, device, storage medium and terminal device
CN110019239A (en) * 2017-12-29 2019-07-16 百度在线网络技术（北京）有限公司 Storage method, device, electronic equipment and the storage medium of reported data
CN110555005A (en) * 2019-08-12 2019-12-10 中国南方电网有限责任公司 Extensible open type paradigm method and device applied to power grid monitoring system

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111913885A (en) * 2020-08-07 2020-11-10 腾讯科技（深圳）有限公司 Log processing method and device, computer readable storage medium and equipment
CN111913885B (en) * 2020-08-07 2021-07-27 腾讯科技（深圳）有限公司 Log processing method and device, computer readable storage medium and equipment
CN113015203A (en) * 2021-03-22 2021-06-22 Oppo广东移动通信有限公司 Information acquisition method, device, terminal, system and storage medium
CN114528258A (en) * 2022-02-18 2022-05-24 北京百度网讯科技有限公司 Asynchronous file processing method, device, server, medium, product and system
CN114528258B (en) * 2022-02-18 2022-12-27 北京百度网讯科技有限公司 Asynchronous file processing method, device, server, medium, product and system

Similar Documents

Publication Publication Date Title
CN107332876B (en) Method and device for synchronizing block chain state
CN111555963B (en) Message pushing method and device, electronic equipment and storage medium
CN111309696A (en) Log processing method and device, electronic equipment and readable medium
CN108900374B (en) Data processing method and device applied to DPI equipment
CN111966289B (en) Partition optimization method and system based on Kafka cluster
CN111538563A (en) Event analysis method and device for Kubernetes
US7814165B2 (en) Message classification system and method
CN108063685B (en) Log analysis method and device
CN113778810A (en) Log collection method, device and system
CN111539206B (en) Method, device, equipment and storage medium for determining sensitive information
CN110955460B (en) Service process starting method and device, electronic equipment and storage medium
CN111159142B (en) Data processing method and device
CN115309907B (en) Alarm log association method and device
CN109284257B (en) Log writing method and device, electronic equipment and storage medium
CN113472858B (en) Buried point data processing method and device and electronic equipment
CN116233164A (en) Method, apparatus, storage medium and processor for collecting device data
CN112749142B (en) Handle management method and system
CN111291127B (en) Data synchronization method, device, server and storage medium
CN110110081B (en) Hierarchical classification processing method and system for mobile internet mass monitoring data
CN114466004A (en) File transmission method, system, electronic equipment and storage medium
CN114422576A (en) Session cleaning method and device, computer equipment and readable storage medium
CN114553944A (en) Early warning message pushing method and system
CN113835905A (en) Message queue load balancing method and device, electronic equipment and medium
CN110661892B (en) Domain name configuration information processing method and device
CN114723349B (en) Daily newspaper sending method, system, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination