CN112199596B - Log filtering processing method, device, equipment and medium - Google Patents

Log filtering processing method, device, equipment and medium Download PDF

Info

Publication number
CN112199596B
CN112199596B CN202011118124.1A CN202011118124A CN112199596B CN 112199596 B CN112199596 B CN 112199596B CN 202011118124 A CN202011118124 A CN 202011118124A CN 112199596 B CN112199596 B CN 112199596B
Authority
CN
China
Prior art keywords
log data
log
filtering
loaded
time
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011118124.1A
Other languages
Chinese (zh)
Other versions
CN112199596A (en
Inventor
方诚杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou DPTech Technologies Co Ltd
Original Assignee
Hangzhou DPTech Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou DPTech Technologies Co Ltd filed Critical Hangzhou DPTech Technologies Co Ltd
Priority to CN202011118124.1A priority Critical patent/CN112199596B/en
Publication of CN112199596A publication Critical patent/CN112199596A/en
Application granted granted Critical
Publication of CN112199596B publication Critical patent/CN112199596B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/953—Querying, e.g. by the use of web search engines
    • G06F16/9535—Search customisation based on user profiles and personalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
    • G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The embodiment of the disclosure provides a method, a device, equipment and a medium for log filtering processing, wherein the method for log filtering processing comprises the following steps: dividing the log file into log data segments; sequentially loading the log data segments into a memory; and filtering each piece of log data of the log data section loaded in the memory to obtain a target log, wherein the target log is log data meeting log filtering conditions. According to the embodiment of the disclosure, the log files are filtered through the segmented loading log, so that the occupation of the process to the memory resource is reduced, the filtering efficiency is improved, and the burden of equipment is smaller.

Description

Log filtering processing method, device, equipment and medium
Technical Field
The embodiment of the disclosure relates to the technical field of information processing, in particular to a method, a device, equipment and a medium for log filtering processing.
Background
In the common network equipment nowadays, a log file recording running information of software and hardware is usually required to be obtained, the specific running condition of the service can be known by looking at the log file of some services, and corresponding measures are carried out. However, the log file usually contains various information and has a huge data volume, so that the user can not obtain the required information conveniently, and therefore, the log file needs to be subjected to certain filtering treatment for use.
In the prior art, when the log is filtered, all log files need to be loaded to a memory for post-processing, so that a large amount of memory is occupied and a large amount of time is consumed.
Disclosure of Invention
In view of this, embodiments of the present disclosure at least provide a method, an apparatus, a device, and a medium for log filtering processing, so as to effectively solve the problem that a large amount of memory resources are occupied when filtering processing logs.
In a first aspect, a method for log filtering processing is provided, the method comprising: dividing the log file into log data segments; sequentially loading the log data segments into a memory; and filtering each piece of log data of the log data section loaded in the memory to obtain a target log, wherein the target log is log data meeting log filtering conditions.
In a second aspect, there is provided an apparatus for log filtering processing, the apparatus comprising: the segmentation module is used for dividing the log file into log data segments; the loading module is used for sequentially loading the log data segments into the memory; the processing module is used for filtering each piece of log data of the log data segment loaded in the memory to obtain a target log, wherein the target log is log data meeting log filtering conditions.
In a third aspect, an electronic device is provided, the device comprising a memory for storing computer instructions executable on the processor for implementing the method of any of the embodiments of the disclosure when the computer instructions are executed.
In a fourth aspect, a computer readable storage medium is provided, on which a computer program is stored, which when executed by a processor implements a method according to any of the embodiments of the present disclosure.
According to the embodiment of the disclosure, the log files are loaded in sections, and each time the log files of the part are loaded, the filtering processing is performed, so that the memory requirement is reduced, the occupation of the filtering processing program on system resources is greatly reduced, the burden of equipment is smaller, and the filtering processing efficiency is improved.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
In order to more clearly illustrate the technical solutions of one or more embodiments of the present disclosure or related technologies, the following description will briefly describe the drawings that are required to be used in the embodiments or related technology descriptions, and it is apparent that the drawings in the following description are only some embodiments described in one or more embodiments of the present disclosure, and other drawings may be obtained according to these drawings without inventive effort to those of ordinary skill in the art.
FIG. 1 is a schematic diagram of a format of a log file stored on disk according to an embodiment of the present disclosure;
FIG. 2 is a flow chart of a method of log filtering processing shown in an embodiment of the present disclosure;
FIG. 3 is a flow chart of another method of log filtering processing shown in an embodiment of the present disclosure;
FIG. 4 is a flow chart of another method of log filtering processing shown in an embodiment of the present disclosure;
FIG. 5 is a flow chart illustrating a method of user-defined filtering rules according to an embodiment of the present disclosure;
FIG. 6 is a block diagram of a log filtering processing device shown in an embodiment of the disclosure;
FIG. 7 is a block diagram of another log filtering processing device shown in an embodiment of the disclosure;
fig. 8 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the disclosure.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present specification. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present description as detailed in the accompanying claims.
The terminology used in the description presented herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the description. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.
Network devices, applications, and operating systems generate a large number of log files on the fly, which are typically stored in text form in memory, typically disk. Fig. 1 shows a format of a log file stored in a disk under a general scenario, where the log file is composed of various pieces of log data, time 1, time 2, and time 3, … … are time attributes of the various pieces of log data, represent time when the various pieces of log data are generated, and column 1, column 2, column 3, and column 4, … … are field data, respectively represent different attributes of the log. For example, when the operating system receives some signals to terminate the running, the content of the process address space and other information about the process state at this time are written into a log file, that is, a core dump file, where column 1 and column 2 may represent the numbers and the positions of the errors respectively. At present, when filtering the log files to screen the needed log information, all the log files need to be loaded into a memory, the log files are often large, and usually have a plurality of G, so that the memory requirement is high when processing the log. Therefore, it is necessary to develop a method for log filtering processing that reduces memory consumption.
As shown in fig. 2, fig. 2 is a flowchart illustrating a method of log filtering processing according to an embodiment of the present disclosure, which may be performed by a device responsible for performing log filtering processing (hereinafter referred to as a filtering processing device), the method may include the steps of:
s101, dividing the log file into log data segments.
In this step, the log file is generated by the network device, the application program and the operating system in operation, or may be downloaded by the cloud end and stored in the memory in text form, and the embodiment does not limit the manner in which the filtering processing device obtains the log file. The memory in the filtering processing device for executing the method of the embodiment is a magnetic disk, and the embodiment does not limit the type of the memory in the device.
The present embodiment is not limited to the manner in which the log file is segmented, such as, but not limited to, the following examples:
for example, the log file may be segmented according to the number of log data, and if the log file has 5 ten thousand log data, the log file may be equally divided into 50 log data segments, each of which has 1 thousand log data. The number of log data pieces included in the divided log file segments may be different, for example.
For another example, the log data may be segmented according to the time attribute of the log data, and the log data generated in each half hour is segmented; for example, log data generated from 2-point half to 3-point may be divided into one segment, and log data generated from 3-point half to 3-point may be divided into another segment.
In the specific implementation, the log data segments can be segmented in other modes, and the principle is that the segmented log data segments cannot overload the memory of the filtering processing equipment, and the log filtering equipment can filter in the memory when performing log filtering.
S102, sequentially loading the log data segments into a memory.
Wherein the sequence can be understood as follows: the log data segment in S101 is not loaded into the memory all at once, but may be divided into multiple loads. For example, the log file is divided into 10 log data segments, and the log data segments can be loaded into the memory 10 times or 5 times, and the specific number of loading segments or loading frequency can be determined according to the processing pressure of the memory, so that the memory does not have too much processing pressure. For example, when the memory idle rate is high at the beginning of processing the log file, a plurality of log data segments are loaded into the memory at a time, and if the memory is occupied by other application programs so that the memory idle rate is low, a plurality of segments can be loaded at a time or only one log data segment can be loaded into the memory at a time. The principle is that the loaded log data segment does not overload the memory.
The loading sequence of each log data segment is not limited in this embodiment, for example, the log data segments may be loaded sequentially according to the time attribute of the log data segment, or the log data segments may be loaded sequentially according to the size of the log data segment.
S103, filtering each piece of log data of the log data segment loaded in the memory to obtain a target log, wherein the target log is log data meeting log filtering conditions.
In the step, after loading the log data segment into the memory, each piece of log data in the log data segment is filtered according to a specified filtering condition, and the remained log data is a target log.
For example, in one embodiment, each piece of log data of a log data segment is filtered using a bpf (berkeley packet filter) expression, where "not" represents a non-operator and "not port 443" represents filtering of a log data piece containing port 443 information; "port 443or port 80" represents filtering log data strips which do not contain any port information of the port 443and the port 80; "and" represents the AND operator, and "port 443and port 80" represents filtering log data strips that do not contain both port 443and port 80 information. For the bpf expression "port 443and (port 138 or port 139 or port 445) and not port 80", it means that only a log data piece satisfying the condition that port 443 information is contained, port 80 information is not contained, and any of ports 138, 139, 445 is contained is left, and the left log data piece is taken as the target log. In addition to using ports as filtering conditions, IP addresses, protocols, and the like may be used as filtering conditions, which are not limited herein.
After the loaded log data segment is filtered, the process returns to S102 to load the log data segment into the memory in order.
According to the log filtering processing method, the log files are loaded in sections, the log files of the parts are loaded each time to carry out filtering processing, the memory requirement is reduced, the occupation of the filtering program to system resources is greatly reduced, the burden of equipment is smaller, and the filtering processing efficiency is improved.
Optionally, in order to further reduce occupation of system process resources, after filtering at least one log data segment to obtain a target log, the target log is sent to a network node.
Wherein the network node may be a remote device. For example, the remote device needs to analyze error information in the log file generated by the software program, after filtering two log data segments in the memory to obtain a target log composed of log data strips containing the required error information, the target log is sent to the remote device needing to further analyze the target log, so as to avoid network congestion and excessive occupation of system process resources.
In an exemplary example, in order to further improve the filtering efficiency and reduce the filtering time, after loading the log data segment into the memory at one time, the filtering process is started to load the log data segment of the next filtering process into the memory at the same time as the loaded log data segment.
For example, in one log loading, the log file is divided into seven log data segments, according to the occupation condition of the memory, three log data segments are loaded from the disk into the memory for the first time, and the filtering process is started for the three log data segments, and at the same time, two log data segments for the second time are continuously loaded into the memory for waiting for the filtering process, wherein the "two log data segments for the second time" is the log data segment for the next filtering process. When the filtering processing of the three log data segments is finished, the preloaded second two log data segments are continuously loaded while the filtering processing is started, namely the rest two log data segments.
The method and the device have the advantages that more time is consumed for loading the log data segments from the disk to the memory, and the scheme in the embodiment avoids the need of waiting for loading the log data segments next time under the condition that the internal memory is fast for filtering the log data segments, so that the filtering efficiency is further improved, and the filtering time is reduced.
As shown in fig. 3, a flowchart of another method of log filtering processing according to an exemplary embodiment is shown, and this embodiment describes a procedure of performing log filtering processing according to a time attribute of log data when the log data at a processing destination point in time is required to be filtered, on the basis of the foregoing embodiment. The "destination time point" may be a designated time when filtering is desired, for example, it is desired to filter logs generated at 8 points and 15 minutes, and then the 8 points and 15 minutes are the destination time points. As shown in fig. 3, the method may include the steps of:
s201, dividing the log file into log data segments.
In this step, the log file is a log file including a destination point in time at which the filtering process is desired.
S202, sequentially loading the log data segments into a memory.
In this step, log data segments are sequentially loaded according to the time attribute of the log data segments, and the specific number of loading segments or loading frequency can be determined according to the processing pressure of the memory.
S203, preloading the log data segment of the next time to the memory.
In this step, the log data segment to be subjected to the filtering process next after the loaded log data segment is preloaded.
S204, comparing whether the time attribute of the last piece of log data of the loaded log data segment is smaller than the destination time point.
In this step, comparing the time attribute of the last log data of the loaded log data segment with the size of the destination time point, if the time attribute of the last log data of the log data segment is smaller than the destination time point, filtering the loaded log data segment is not performed, and returning to step S203; if not, the next step S205 is performed.
S205, filtering each piece of log data of the log data segment loaded in the memory to obtain a target log.
As shown in fig. 4, a flowchart of another method of log filtering processing according to an exemplary embodiment is shown, and this embodiment describes a procedure of performing log filtering processing according to a time attribute of log data when the log data of a processing destination period is required to be filtered, on the basis of the foregoing embodiment. The "destination time period" may be a specified period of time for which filtering is desired, for example, it is desired to filter logs generated from 8 points to 8 points by 15 minutes, and then 15 minutes from 8 points to 8 points by 15 minutes is the destination time period. As shown in fig. 4, the method may include the steps of:
s201a, dividing the log file into log data segments.
In this step, the log file is a log file containing a destination time period for which filtering processing is desired.
S202a, sequentially loading the log data segments into a memory.
In this step, log data segments are sequentially loaded according to the time attribute of the log data segments, and the specific number of loading segments or loading frequency can be determined according to the processing pressure of the memory.
S203a, preloading the log data segment of the next time to the memory.
In this step, the log data segment to be subjected to the filtering process next after the loaded log data segment is preloaded.
S204a, comparing whether the time attribute of the last piece of log data of the loaded log data segment is smaller than the time at the beginning of the destination time segment.
In this step, comparing the time attribute of the last log data of the loaded log data segment with the time at the beginning of the destination time segment, if the time attribute of the last log data of the log time segment is less than the time at the beginning of the destination time segment, filtering the loaded log data segment, and returning to step S203a; if not, the next step S205a is performed.
S205a, filtering each piece of log data of the log data segment loaded in the memory to obtain a target log.
According to the embodiment of the disclosure, under the condition that a user only needs log data of a certain time point/data segment, each piece of log data of the log file is not required to be filtered, but the time point or the time segment to be queried is compared with the time attribute of the last piece of log data of the loaded log data segment, and the comparison result shows that the log data to be queried is not in the loaded log data segment, the filtering processing of the log data segment is skipped, so that the filtering time is greatly reduced, and the filtering efficiency is improved.
In another embodiment, in the log filtering processing method of the embodiment of the present disclosure, a user may further customize filtering conditions during filtering processing of each piece of log data, as shown in fig. 5, the steps are as follows:
s301, acquiring a user-defined semantic rule.
In this step, the user defines the sign of the semantic rule, where the semantic rule includes logical operations such as "and", "or", "not", "nand", "nor", "exclusive or", "same or", and the user can select the logical operation and assign the sign according to the filtering requirement and the custom grammar.
For example, the user selects "and", "or", "not" as the logical operation to be used in filtering, and "a" may be selected to represent "and", "B" or "," C "to represent" not ", and the user-selectable characters and logical operations are not limited herein. Wherein, the AND is a binary operator, two conditions are judged, if both conditions are true, the true is returned, otherwise, the false is returned; the OR symbol is also a binary operator, two conditions are judged, if both conditions are false, the false is returned, and otherwise, the true is returned; the NOT symbol is a unitary operator that determines a condition, returns false if the condition is true, and returns true if the condition is false.
S302, obtaining a filtering expression which is input by a user and is determined according to the semantic rule.
In this step, a filtering expression input by the user is obtained, and the symbols in the filtering expression are user-defined symbols "a", "B" and "C" of "and", "or", "not".
For example, the user inputs "port 443A port 80" indicating that a log data bar containing information of both port 443and port 80 is acquired, and other log data which does not meet the condition is filtered; user input "port 443B port 80" means that a log data bar containing any one port information of the port 443and the port 80 is acquired, and other log data which does not meet the condition are filtered; user input "C ICMP" means that the log data bar with ICMP (network control message protocol) is filtered, and other log data meeting the condition is left; user input "host 192.168.0.1A C port 80A (port 138B port 139B port 445)" means that the device that acquired the IP address 192.168.0.1 contains any of port information 138, 139, and 445 and does not contain port 80 information, and other unsatisfied log data is filtered. The filtering conditions such as the above port, protocol, IP address, and the like, and the filtering conditions such as inflow and outflow are not limited thereto.
The method for obtaining the semantic rule or the filtering expression determined by the user by using the filtering processing device in the embodiment of the disclosure can be in various manners. For example, the filtering processing device may provide a man-machine interaction interface through which a user may input or select a symbol, symbol meaning, etc. that the user wants to use. For another example, the filtering processing device obtains a rule definition file loaded by the user, and obtains the rule or the filtering expression determined by the user by parsing the rule definition file.
After the filter expression determined by the user is obtained, the filtering processing of the log data may be performed according to the filter expression, and the specific filtering processing manner may be referred to the foregoing embodiment, which is not described herein.
In this embodiment, the filtering expression input by the user is used to filter each piece of log data of the log data segment loaded in the memory, so as to obtain the log data meeting the condition of the filtering expression as the target log, and the filtering grammar has high degree of freedom, so that the customization requirement of the user can be met.
Corresponding to the foregoing method embodiment, the present disclosure further provides an apparatus for log filtering processing, as shown in fig. 6, where the apparatus may include:
a segmentation module 401 for dividing the log file into log data segments;
a loading module 402, configured to sequentially load the log data segments into a memory;
the processing module 403 is configured to perform filtering processing on each piece of log data of the log data segment loaded in the memory, so as to obtain a target log, where the target log is log data that meets a log filtering condition.
The device for filtering the log in the embodiment of the disclosure loads the log files in a segmented mode, and filters the log files of a part every time without occupying too much memory, so that the efficiency of filtering is improved.
As shown in fig. 7, fig. 7 is a block diagram of another log filtering processing apparatus according to an exemplary embodiment of the present disclosure, where the embodiment may further include, on the basis of the foregoing apparatus shown in fig. 6:
the definition module 400 is used for acquiring a user-defined semantic rule and acquiring a filtering expression input by a user and determined according to the semantic rule;
and the sending module 404 is configured to send the target log to the network node after each filtering process finishes at least one log data segment to obtain the target log.
The loading module 402 may further include a preloading module 4021, configured to start filtering the log data segment loaded after loading the log data segment into the memory at one time, and simultaneously load the log data segment filtered next time into the memory.
The processing module 403 may further include:
a time point processing submodule 4031, configured to compare, when log data at a destination time point is to be filtered, a time attribute of last log data of the loaded log data segment with a size of the destination time point after each log data segment is loaded, and if the time attribute of last log data of the log data segment is smaller than the destination time point, not to filter the loaded log data segment;
a time period processing submodule 4032, configured to compare, when log data of a destination time period is to be filtered, a time attribute of last log data of the loaded log data period with a time at the beginning of the destination time period after each log data period is loaded, and if the time attribute of last log data of the log time period is less than the time at the beginning of the destination time period, not to filter the loaded log data period;
and a filtering submodule 4033, configured to perform filtering processing on each piece of log data of the log data segment loaded in the memory by using the filtering expression, so as to obtain log data that meets the condition of the filtering expression, as a target log.
According to the log filtering device provided by the embodiment, through a segmented loading and preloading mechanism, the log data segment to be processed next time is preloaded during filtering processing, so that the occupation of a filtering program to system resources is greatly reduced; the log data after the filtering treatment is timely sent, and network blockage is prevented; the time attribute of the log data is utilized, so that the times of filtering processing are greatly reduced when the log data of certain time points or time periods are searched, and the filtering efficiency is improved; the user can customize the filtering grammar, and the filtering expression is more conveniently used.
The implementation process of the functions and roles of each module in the above device is specifically shown in the implementation process of the corresponding steps in the above method, and will not be described herein again.
As shown in fig. 8, fig. 8 is a schematic hardware structure of an electronic device according to an embodiment of the present disclosure, where the electronic device includes a memory 501, a processor 502, a memory 503, and a network interface 504, where the memory 501 is configured to store computer instructions and log files that can be executed on the processor, the processor 502 is configured to implement a method for log filtering processing according to any embodiment of the present disclosure when the computer instructions are executed, the memory 503 is configured to temporarily store the computer instructions and data, and the network interface 504 is configured to exchange data with an external device.
The present disclosure also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a method of log filtering processing according to any of the embodiments of the present disclosure.
For the device embodiments, reference is made to the description of the method embodiments for the relevant points, since they essentially correspond to the method embodiments. The apparatus embodiments described above are merely illustrative, wherein the modules illustrated as separate components may or may not be physically separate, and the components shown as modules may or may not be physical, i.e., may be located in one place, or may be distributed over a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purposes of the present description. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
Other embodiments of the present description will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This specification is intended to cover any variations, uses, or adaptations of the specification following, in general, the principles of the specification and including such departures from the present disclosure as come within known or customary practice within the art to which the specification pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the specification being indicated by the following claims.
It is to be understood that the present description is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.
The foregoing description of the preferred embodiments is provided for the purpose of illustration only, and is not intended to limit the scope of the disclosure, since any modifications, equivalents, improvements, etc. that fall within the spirit and principles of the disclosure are intended to be included within the scope of the disclosure.

Claims (7)

1. A method of log filtering processing, the method comprising:
dividing the log file into log data segments;
sequentially loading the log data segments into a memory, including: after loading the log data segment into the memory once, starting to filter the loaded log data segment and loading the log data segment subjected to the next filtering process into the memory at the same time;
filtering each piece of log data of the log data section loaded in the memory to obtain a target log, wherein the target log is log data meeting log filtering conditions;
before the filtering processing is performed on each piece of log data of the log data segment loaded in the memory, the method further comprises:
when the log data of a target time point is to be filtered, after a log data section is loaded, comparing the time attribute of the last log data of the loaded log data section with the size of the target time point, and if the time attribute of the last log data of the log data section is smaller than the target time point, not filtering the loaded log data section; and/or the number of the groups of groups,
when the log data of the destination time period is to be filtered, after each log data segment is loaded, comparing the time attribute of the last log data of the loaded log data segment with the time of the beginning of the destination time period, and if the time attribute of the last log data of the log time period is smaller than the time of the beginning of the destination time period, not filtering the loaded log data segment.
2. The method according to claim 1, wherein the method further comprises:
and after each filtering process is completed on at least one log data segment to obtain a target log, sending the target log to a network node.
3. The method according to claim 1, wherein the method further comprises:
acquiring a user-defined semantic rule;
acquiring a filtering expression input by a user and determined according to the semantic rule;
the filtering processing is performed on each piece of log data of the log data segment loaded in the memory to obtain a target log, including:
and filtering each piece of log data of the log data segment loaded in the memory through the filtering expression to obtain the log data meeting the condition of the filtering expression as a target log.
4. An apparatus for log filtering processing, the apparatus comprising:
the segmentation module is used for dividing the log file into log data segments;
the loading module is used for sequentially loading the log data segments into the memory, and is specifically used for: after loading the log data segment into the memory once, starting to filter the loaded log data segment and loading the log data segment subjected to the next filtering process into the memory at the same time;
the processing module is used for filtering each piece of log data of the log data segment loaded in the memory to obtain a target log, wherein the target log is log data meeting log filtering conditions;
the processing module further comprises at least one sub-module of:
the time point processing sub-module is used for comparing the time attribute of the last log data of the loaded log data segment with the size of the target time point after loading one log data segment when the log data of the target time point is to be filtered, and if the time attribute of the last log data of the log data segment is smaller than the target time point, the loaded log data segment is not filtered;
and the time segment processing submodule is used for comparing the time attribute of the last log data of the loaded log data segment with the time of the beginning of the destination time segment after loading one log data segment when the log data of the destination time segment is to be filtered, and if the time attribute of the last log data of the log time segment is smaller than the time of the beginning of the destination time segment, the loaded log data segment is not filtered.
5. The apparatus of claim 4, wherein the apparatus further comprises:
the definition module is used for acquiring user-defined semantic rules and acquiring filtering expressions which are input by a user and are determined according to the semantic rules;
the processing module comprises:
and the filtering sub-module is used for filtering each piece of log data of the log data segment loaded in the memory through the filtering expression to obtain log data meeting the condition of the filtering expression as a target log.
6. An electronic device comprising a memory, a processor for storing computer instructions executable on the processor for implementing the method of any one of claims 1 to 3 when the computer instructions are executed.
7. A computer readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method of any of claims 1 to 3.
CN202011118124.1A 2020-10-19 2020-10-19 Log filtering processing method, device, equipment and medium Active CN112199596B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011118124.1A CN112199596B (en) 2020-10-19 2020-10-19 Log filtering processing method, device, equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011118124.1A CN112199596B (en) 2020-10-19 2020-10-19 Log filtering processing method, device, equipment and medium

Publications (2)

Publication Number Publication Date
CN112199596A CN112199596A (en) 2021-01-08
CN112199596B true CN112199596B (en) 2023-08-25

Family

ID=74008778

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011118124.1A Active CN112199596B (en) 2020-10-19 2020-10-19 Log filtering processing method, device, equipment and medium

Country Status (1)

Country Link
CN (1) CN112199596B (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105426292A (en) * 2015-10-29 2016-03-23 网易（杭州）网络有限公司 Game log real-time processing system and method
CN106294866A (en) * 2016-08-23 2017-01-04 北京奇虎科技有限公司 A kind of log processing method and device
CN106503008A (en) * 2015-09-07 2017-03-15 网宿科技股份有限公司 File memory method and device and file polling method and apparatus
CN108876508A (en) * 2018-05-03 2018-11-23 上海海事大学 A kind of electric business collaborative filtering recommending method
CN109446174A (en) * 2018-10-30 2019-03-08 东软集团股份有限公司 Logdata record method, apparatus and computer readable storage medium
CN109977089A (en) * 2019-03-13 2019-07-05 深圳壹账通智能科技有限公司 Blog management method, device, computer equipment and computer readable storage medium
CN110209643A (en) * 2019-04-23 2019-09-06 深圳壹账通智能科技有限公司 A kind of data processing method and device
CN110362450A (en) * 2019-07-16 2019-10-22 深圳市网心科技有限公司 A kind of log data acquisition method, device and computer readable storage medium
CN110399350A (en) * 2018-04-19 2019-11-01 沪江教育科技（上海）股份有限公司 Processing method, device, storage medium and the electronic equipment of log information

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106503008A (en) * 2015-09-07 2017-03-15 网宿科技股份有限公司 File memory method and device and file polling method and apparatus
CN105426292A (en) * 2015-10-29 2016-03-23 网易（杭州）网络有限公司 Game log real-time processing system and method
CN106294866A (en) * 2016-08-23 2017-01-04 北京奇虎科技有限公司 A kind of log processing method and device
CN110399350A (en) * 2018-04-19 2019-11-01 沪江教育科技（上海）股份有限公司 Processing method, device, storage medium and the electronic equipment of log information
CN108876508A (en) * 2018-05-03 2018-11-23 上海海事大学 A kind of electric business collaborative filtering recommending method
CN109446174A (en) * 2018-10-30 2019-03-08 东软集团股份有限公司 Logdata record method, apparatus and computer readable storage medium
CN109977089A (en) * 2019-03-13 2019-07-05 深圳壹账通智能科技有限公司 Blog management method, device, computer equipment and computer readable storage medium
CN110209643A (en) * 2019-04-23 2019-09-06 深圳壹账通智能科技有限公司 A kind of data processing method and device
CN110362450A (en) * 2019-07-16 2019-10-22 深圳市网心科技有限公司 A kind of log data acquisition method, device and computer readable storage medium

Also Published As

Publication number Publication date
CN112199596A (en) 2021-01-08

Similar Documents

Publication Publication Date Title
US7681196B2 (en) Providing optimal number of threads to applications performing multi-tasking using threads
CN112506950A (en) Data aggregation processing method, computing node, computing cluster and storage medium
CN117238433B (en) Method for automatically isolating document data based on Libreoffice
CN111338628B (en) Component rendering method and device
CN112199596B (en) Log filtering processing method, device, equipment and medium
CN117763511A (en) Method and device for operating hook function, computer equipment and storage medium
CN110764705B (en) Data reading and writing method, device, equipment and storage medium
CN113296962B (en) Memory management method, device, equipment and storage medium
CN112764897B (en) Task request processing method, device and system and computer readable storage medium
US20170090820A1 (en) Method and device for operating a many-core system
CN115185576A (en) Process customization method and system based on process application system
CN111079390B (en) Method and device for determining selection state of check box list
CN114238250A (en) Program file compression method and device and electronic equipment
CN111045787B (en) Rapid continuous experiment method and system
WO2016131242A1 (en) Method and device for file parsing
US8788483B2 (en) Method and apparatus for searching in a memory-efficient manner for at least one query data element
CN105988866B (en) Information processing unit and its control method
CN106934555B (en) Workflow organization method and device, computer equipment and storage medium
CN111381977A (en) Message processing method and device
CN117519605A (en) Read-write efficiency improving method and related equipment
CN108958654B (en) Management method and related device of storage system
CN117076004B (en) Micro-service packaging and merging method and device and electronic equipment
CN112148384B (en) SCA waveform component loading method and device, readable medium and electronic equipment
CN115277140B (en) Policy configuration, risk identification method and device, readable storage medium and computing device
CN114884974B (en) Data multiplexing method, system and computing device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant