CN116489005A - Log service system and log processing method - Google Patents
Log service system and log processing method Download PDFInfo
- Publication number
- CN116489005A CN116489005A CN202310415428.1A CN202310415428A CN116489005A CN 116489005 A CN116489005 A CN 116489005A CN 202310415428 A CN202310415428 A CN 202310415428A CN 116489005 A CN116489005 A CN 116489005A
- Authority
- CN
- China
- Prior art keywords
- log
- logs
- module
- user
- distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 47
- 238000001914 filtration Methods 0.000 claims abstract description 26
- 230000011218 segmentation Effects 0.000 claims description 24
- 238000000034 method Methods 0.000 claims description 18
- 238000004458 analytical method Methods 0.000 claims description 9
- 230000005856 abnormality Effects 0.000 claims description 3
- 230000008569 process Effects 0.000 description 7
- 239000012634 fragment Substances 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000004590 computer program Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008520 organization Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 230000000739 chaotic effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000013467 fragmentation Methods 0.000 description 1
- 238000006062 fragmentation reaction Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000001681 protective effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/02—Capturing of monitoring data
- H04L43/028—Capturing of monitoring data by filtering
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application provides a log service system and a log processing method, wherein the log service system comprises: the collecting module is used for receiving the logs sent by the network equipment through the distributed collecting components contained in the collecting module; the processing module is used for acquiring the logs collected by the collecting module, providing the acquired logs to a custom component contained in the processing module, and filtering the acquired logs by the custom component according to personalized filtering rules set by a user to generate a result log set; and the storage module is used for acquiring the result log set and storing the result log set through a distributed storage component contained in the storage module.
Description
Technical Field
The present disclosure relates to the field of log service technologies, and in particular, to a log service system and a log processing method.
Background
In network devices such as routers and switches, a manager can master the system status at any time by checking the system records, and the system logs record the size events occurring at any time in the system.
In the related art, the network device may transmit log information to the remote server in a user data protocol manner. However, in a high availability architecture, the amount of logs increases as the traffic grows, and when the traffic reaches a certain scale, the architecture becomes complex, the logs become more cluttered, and viewing the specified logs becomes inefficient. Therefore, there is a need for a log service system and log management method capable of efficiently processing a large number of logs.
Disclosure of Invention
In view of this, the present application provides a log service system and a log processing method, so as to solve the defects existing in the related art, and the technical scheme of the present application is as follows:
according to an embodiment of the first aspect of the present application, there is provided a log service system, including:
the collecting module is used for receiving the logs sent by the network equipment through the distributed collecting components contained in the collecting module;
the processing module is used for acquiring the logs collected by the collecting module, providing the acquired logs to a custom component contained in the processing module, and filtering the acquired logs by the custom component according to personalized filtering rules set by a user to generate a result log set;
and the storage module is used for acquiring the result log set and storing the result log set through a distributed storage component contained in the storage module.
Optionally, the processing module is further configured to:
and analyzing the logs collected by the collection module or the logs contained in the result log set by the user-defined component according to the personalized analysis rule set by the user to obtain the logs in the user-specified format.
Optionally, the collecting module is further configured to: and sending the collected logs to the processing module through a distributed message queue contained in the collecting module.
Optionally, the collecting module is further configured to: storing the collected logs to a disk for backup;
the processing module is further configured to: and in the case of abnormality of the system, reading the log collected by the collection module from a disk.
Optionally, the log service system further includes:
and the search module is used for responding to the request sent by the search client, determining the log meeting the search condition from the logs stored in the storage module according to the search condition carried in the request, and returning to the search client.
Optionally, the storage module is further configured to:
based on a preset word segmentation rule, segmenting each log in the result log set;
calculating a weight value for each log in the result log set based on a preset weight rule;
storing each log and the word segmentation result and the weight value corresponding to each log;
the search module is further configured to: and sorting the logs meeting the search conditions of the user according to the word segmentation result and the weight value corresponding to each log.
Optionally, the log service system further includes:
the organizing module is used for acquiring logs meeting the search conditions of the user, organizing the acquired logs according to the display mode carried in the request sent by the search client, and returning the organized logs to the search client so that the search client displays the logs according to the display mode.
Optionally, the distributed collection component receives logs sent by the network device through a plurality of pipes, and the plurality of pipes run in independent threads respectively.
Optionally, the distributed collection component includes: logstash, flume, filecoat;
the distributed storage component includes: elasticsearch, mysql, oracle;
the distributed message queue includes: kafka, activeMQ, rabbitMQ, rockerMQ.
According to an embodiment of a second aspect of the present application, there is provided a log processing method applied to a log service system, where the log service system includes a distributed collection component, a custom component, and a distributed storage component, and the method includes:
receiving a log sent by the network equipment through the distributed collection component;
the user-defined component filters the collected logs according to the user-defined personalized filtering rules to generate a result log set;
and storing the result log set through the distributed storage component.
In the technical scheme provided by the application, the log service system can collect logs of a plurality of different network devices in the network architecture through the distributed collection assembly in the collection module, so that the user-defined assembly in the processing module can carry out uniform filtering processing on the collected logs of different data sources. The filtering rules applied in the custom component are personalized filtering rules set by the user according to the requirements, useless logs for the user can be filtered through processing, log information is simplified, further, the distributed storage component in the storage module only stores the logs meeting the requirements of the user, resources required for storage are saved, query efficiency of the user on the logs in subsequent work is improved, and user experience is improved. Meanwhile, the distributed collection assembly and the distributed storage assembly can be horizontally expanded along with the increase of the architecture scale, can support collection and storage work of a large number of logs, and ensures the performance of a log service system.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the disclosure.
Drawings
In order to more clearly illustrate the embodiments of the present description or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments described in the embodiments of the present description, and other drawings may be obtained according to these drawings for a person having ordinary skill in the art.
FIG. 1 is a schematic diagram of a log service system module according to an exemplary embodiment of the present disclosure;
FIG. 2 is a log processing flow diagram of a log service system according to an exemplary embodiment of the present disclosure;
FIG. 3 is a flow chart of a log processing method according to an exemplary embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a log processing device according to an exemplary embodiment of the present disclosure;
fig. 5 is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present specification. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present description as detailed in the accompanying claims.
The terminology used in the description presented herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the description. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information, without departing from the scope of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.
The system log is information for recording hardware, software and system problems in the system, such as network equipment, system, service program, etc., and generates a log-called event record when in operation, wherein the log-called event record comprises a plurality of lines of logs, each line of logs records the description of the date, time, user, action, etc., and the event occurring in the system can be monitored. The user can check the system records through the system log, grasp the system condition at any time, for example, check the cause of error occurrence or find the trace left by an attacker when being attacked.
Currently, syslog is an industry-standard protocol that can be used to log network devices, is supported by almost all network devices, and is capable of logging log messages of multiple event types. Common to Syslog enabled devices are routers, switches, printers, etc., and even UNIX system servers can support the generation of Syslog messages to record user login, firewall events, apache or nmginx Access logs, etc.
Today network devices commonly support the Syslog protocol, i.e., the system log protocol, by which almost all network devices can communicate log information to a remote server in a user data protocol. Meanwhile, there are some system log servers, such as Syslogd, rsyslogd, syslog-ng, kiwisyslog, which can simply collect system logs in network devices.
However, in a high availability architecture, single point mode is often the greatest risk of high availability of the system, and should be avoided during system design. Therefore, in order to ensure that the system is highly available, the architecture design should conform to redundant standard measurement, but this also results in that the logs in the highly available architecture are generally scattered on multiple nodes in the architecture, the log amount will increase with the increase of the service, when the service reaches a certain scale, the architecture becomes complex, the logs will be more chaotic, and the log amount is very large. In this case, the ordinary system log server cannot bear load, resulting in performance degradation or even crash, and meanwhile, in the collected massive log information, the efficiency of checking the specified log is greatly reduced.
In order to solve the problems, the application provides a log service system, wherein a collection module in the system comprises a distributed collection assembly, and a storage module comprises a distributed storage assembly, so that a large amount of logs generated in a network architecture can be collected and stored, and the log service system can be horizontally expanded along with the increase of the architecture scale, so that the performance of the log service system is ensured; meanwhile, the user-defined assembly is adopted to carry out personalized filtering treatment on the collected logs, only logs required by a user are reserved and stored, resources required by storage are saved, query efficiency of the user on the logs in subsequent work is improved, and user experience is improved. Next, embodiments of the present application will be described in detail.
FIG. 1 is a schematic diagram of a log service system according to an exemplary embodiment of the present application, which may include:
a collecting module 11, configured to receive, through a distributed collecting component included in the collecting module, a log sent by a network device;
the processing module 12 is configured to obtain the log collected by the collecting module, provide the obtained log to a custom component included in the processing module, and filter the obtained log by the custom component according to a personalized filtering rule set by a user to generate a result log set;
and the storage module 13 is used for acquiring the result log set, and storing the result log set through a distributed storage component contained in the storage module.
For a detailed description of the log service system of the present application, an exemplary log processing flow will be described with reference to fig. 2.
In an exemplary embodiment, the log service system is disposed on one or more servers, and the system logs transmitted from different network devices are received through distributed collection components included in the collection module 11 at designated ports of the servers. In the selection of the collection mode, each network device in the architecture can actively push the logs to the log service system, or the collection module 11 can collect the logs uniformly. The network device in the present application is a physical entity connected to a network, and the network device may be a router, a switch, a printer, or other servers, such as an e-commerce platform, a data monitoring server, or the like, which is not limited in this application.
Under the condition that the whole architecture is simpler, if the network equipment needing to monitor the activity is fewer and the quantity of collected logs is smaller, the distributed collection components in the collection module 11 can be arranged on a single server or service node, and at the moment, one server or service node can support the log collection work of the whole architecture, so that the cost of the server can be saved. Under the condition that the whole architecture is complex, if the network equipment needing to monitor activities is more, or the service quantity in the architecture is increased and the service complexity is increased, so that the collected log quantity is greatly increased, the distributed collection assembly in the collection module 11 can be expanded to a plurality of servers or a plurality of service nodes to work, namely, the cluster is horizontally expanded, the pressure of the existing service is dispersed, the collection work of a large number of logs can be supported, meanwhile, the work of the distributed collection assembly in the existing servers or the service nodes is not influenced during expansion, and the log collection efficiency is further improved. Here, logstash, flume, filebat may be selected as the distributed collection component, or other collection components supporting distributed collection components may be selected, which is not limited in this application.
In an exemplary embodiment, the collecting module 11 further includes a distributed message queue, and the collected logs are sent to the processing module 12. On the one hand, the distributed message queue can receive a large amount of logs collected by the distributed collection assembly, and can be horizontally expanded due to the distributed characteristic, so that the system can cope with higher load and larger data volume, and meanwhile, when some nodes in the cluster fail, other nodes can continuously support work, the whole system cannot be influenced, and better fault tolerance is achieved. On the other hand, the distributed message queue can be used as a buffer zone, and because the distributed message queue adopts a message transmission mechanism based on asynchronous communication, asynchronous communication can be carried out among different programs, so that log data transmission between a sender and a receiver is not needed at the same time, the coupling between components is reduced, the elasticity of the components is improved, and the distributed message queue can play a good buffering role under the condition that the collected log quantity suddenly increases. It should be noted that the distributed message queue included in the flow shown in fig. 2 is an optional component. Here, kafka, activeMQ, rabbitMQ, rockerMQ may be selected as a distributed message queue, or other message queue frames may be selected, or a simple queue structure may be used for message processing, which is not limited in this application.
In an exemplary embodiment, the processing module 12 obtains the log collected by the collection module 11 and provides it to custom components within the processing module 12. The user-defined component comprises personalized filtering rules set according to the requirements of users, the collected logs are filtered through the rules, the logs obtained after the filtering are used as result logs, and a result log set is generated. The user refers to a user of the log service system, that is, a person who needs to perform a series of operations such as checking, searching or analyzing the log sent by the network device, and the custom component is a component custom developed according to the requirement of the user. When the custom component is developed, the standard of the distributed component can be followed, so that the custom component can be horizontally expanded according to actual needs in subsequent use and applied to a plurality of servers or service nodes. The personalized filtering rules can be determined together when the custom component is developed, and can be directly written into logic codes of the custom component, or the interface is reserved in the custom component without writing the logic codes, and the filtering rules are set and changed through the interface. In this way, the personalized filtering rules can be changed more flexibly, and can be easily adjusted when the filtering requirements of the user change.
In an exemplary embodiment, the filter criteria may include information for each dimension of the log, such as time, source, keywords, format, etc. of the log. For example, in an architecture including A, B, C, D four network devices, each running its own service and generating corresponding logs, such as subsystem initialization records, request execution records, etc., the distributed collection component included in the collection module 11 will collect all logs generated in the four devices. If the user only needs to pay attention to log information about warnings, errors in network devices a and C in the architecture, the filter conditions can be set to: time [ unlimited ]; source [ A, C ], keyword [ Warning, error ], format [ not limited ]. The custom component filters the collected logs with this filtering rule, which will result in all logs from network devices a and C regarding alerts and errors at all times. Because all of the logs of network device A, D and other logs in network device A, C are useless logs to the user, the user does not need to pay attention to these log information nor need to store them. And taking the filtered logs as result logs, generating a result log set, taking the logs which are reserved after the filtering as the result logs, and generating a result log set, wherein all the logs in the result log set are all the logs required by the user.
It should be noted that, in the actual use process, the filtered logs may be processed according to needs, for example, the filtered logs may be directly deleted to save storage space, or stored in a common disk for a certain period of time, and deleted after the retention period is exceeded, so that a user can check related information of other recent logs when checking problems according to the logs. The application is not limited to a particular manner of processing the filtered logs.
In an exemplary embodiment, custom components in the processing module 12 may also parse the log. Similar to the personalized filtering rules, the personalized parsing rules in the custom component are set according to the requirements of the user. The user can select a log analysis method, such as clustering, frequent item mining and the like, and analyze the log into a format designated by the user through the analysis method selected by the user so as to improve the processing efficiency of subsequent work. Here, the log parsing by the custom component included in the flow shown in fig. 2 is an optional step, and the log to be parsed may be a log collected by the collecting module 11, or may be a log in a result log set generated after being processed by the processing module 12, and may be a partial log in the log. For example, the logs collected by the collecting module 11 include Debug, info, warning, fatal and other types of logs, and after being filtered by the custom component of the processing module 12, the logs of the Info, warning, fatal type are reserved, wherein the user has low attention to the logs of the Info type and is not commonly used, only focuses on the logs of the Fatal type and frequently checks and searches in the subsequent work, and then the custom component can analyze the logs of the Fatal type. Assuming that the user analyzes the logs by adopting a clustering method, the Fatal logs with high similarity can be aggregated into one type by calculating the similarity among log texts, and the common modes of the Fatal logs are extracted. Specifically, the feature extraction can be performed on the logs based on natural language processing, then the logs are clustered by using the similarity of log texts, and the log templates are mined, so that the Fatal type logs in different network devices are unified in format. In the clustering analysis process, a user can determine the clustering times according to the demands, the more the clustering times are, the higher the concentration degree of the mode is, for example, the original 5 Fatal logs from different network devices are clustered for the first time to obtain 2 log formats, and clustered for the second time to obtain 1 log format, namely the final log format. Meanwhile, before the next clustering, the user can also decide which log format to select for later analysis by himself, for example, the 5 Fatal logs obtain 2 log formats after the first clustering, and the user can manually select one of the formats, so that the format is used as the standard when the next clustering is performed, and the log result of the format which finally meets the requirements of the user is obtained. Through log analysis, the original logs can be unified to the same format, so that the efficiency in subsequent searching is improved, the user can check related log information more conveniently, the user can develop subsequent work, and the user experience is improved.
It should be noted that, the setting mode of the personalized parsing rule is the same as that of the personalized filtering rule, and the personalized parsing rule can be determined together when the custom component is developed, and can be directly written into the logic code of the custom component, or an interface can be reserved in the custom component, and the parsing rule is set and changed through the interface. In this way, the personalized resolution rules can be changed more flexibly, and can be easily adjusted when the resolution requirements of the user change.
In an exemplary embodiment, the storage module 13 obtains the result log set generated by the processing module 12, and stores the result log set through a distributed storage component included in the storage module 13. Similar to the aforementioned distributed collection components, in the case of a simpler overall architecture, the distributed storage components in the storage module 13 can be arranged on a single server or service node to save server costs; under the condition that the whole architecture is complex, the distributed storage components in the storage module 13 can be expanded to a plurality of servers or a plurality of service nodes to work, namely, the cluster is horizontally expanded, the pressure of the existing service is dispersed, the storage work of a large number of logs can be supported, and meanwhile, the work of the distributed storage components in the existing servers or the service nodes is not influenced during expansion.
In practical use, the distributed storage component may store the log in a primary and secondary sharding manner. The elastiscearch data analysis engine can be used as a distributed storage component, and other components supporting distributed storage, such as Mysql, oracle or file mode storage, can be used. For example, when using an elastic search for storage, each index may be divided into multiple primary slices, each with several copies, i.e., secondary slices. The main fragments are responsible for the read-write operation of data, and the auxiliary fragments are used for data redundancy and improving the search performance. When a primary shard fails for various reasons, the distributed storage component automatically selects one of the corresponding duplicate shards as a new primary shard and broadcasts this information to other nodes. Because each fragment is independently stored, the fragments can be distributed to different nodes, the horizontal expansion of the data is realized, when a part of servers in the cluster have problems or a certain fragment fails, the data can be read from other fragments, the availability of the log data is ensured, and the reliability of the log service system is improved.
It should be noted that, the distributed storage component may also include other functions besides the above storage by the primary and secondary fragmentation, for example, master node (Master) election and data recovery, and a technician may set a working mode of the distributed storage component according to actual needs.
According to the log service system, the collection module comprises the distributed collection assembly, and the storage module comprises the distributed storage assembly, so that a large amount of logs generated in a network architecture can be collected and stored, and the logs can be horizontally expanded along with the increase of the architecture scale, so that the load demand and the service performance are ensured; meanwhile, the user-defined assembly is adopted to conduct personalized processing on the collected logs, only logs required by a user are reserved and stored, resources required by storage are saved, query efficiency of the user on the logs in subsequent work is improved, and user experience is improved.
In an exemplary embodiment, the collecting module 11 may store the collected log to a disk as a backup, and the processing module 12 may read the collected log from the disk in case of an abnormality of the log service system. In another exemplary embodiment, the collecting module 11 may wait for the system to recover, read the collected log from the disk, and send the log to the processing module 12 normally. The collection module 11 persists the collected log to disk and ensures the ordering of log data during data recovery after a system failure. For example, the log service system fails at time a, recovers at time B, and when the processing module of the system resumes operation, the log collected before time a is still preferentially processed. By adopting the method, the reliability of the log service system can be improved, and meanwhile, the log data can be ensured to be processed in time to a certain extent, so that the influence on the subsequent work of the log service system is avoided.
In an exemplary embodiment, the distributed collection components in collection module 11 collect logs via pipes, which run independently in independent threads. In the pipeline mode, logs are processed in parallel in the collecting process, and compared with single threads, the multi-thread pipeline can fully utilize the computing capacity of a multi-core CPU (Central Processing Unit ) and improve the processing speed. In the case where a large number of logs need to be collected, the use of pipeline parallel collection of logs can significantly reduce collection time.
In an exemplary embodiment, the log service system of the present application further includes a search module 14, capable of responding to a request sent by a search client, determining, according to a search condition carried in the request, a log meeting the search condition from the logs stored in the storage module, and returning to the search client. The search client is a client used by a user, can be a platform with a graphical interface, and can also be a server only in a form of a command line. The user can send a search request to the log service system through the search client, and the search request needs to carry search conditions for indicating logs expected by the user. The search condition may include information of each dimension of the log, such as time, source, included information, keywords, etc. of the log, and may be limited to a single condition for searching, or may be combined. For example, if the search condition in the search request sent by the user is { date:2023-01-01}, the search module 14 searches all the logs with the date 2023, month 1 and day 1 in the logs stored in the storage module 13 and returns; if the search condition in the search request sent by the user is { date:2023-01-01; type: error }, then the search module 14 looks up the full log of the date 2023, month 1, and type Error, and returns to the search client. It should be noted that the search module included in the flow shown in fig. 2 is an optional module, and a technician can determine whether to use according to the requirement. In the log service system of the embodiment of the application, the search module 14 enables the user to use the log service system more conveniently, the log required by the user is accurately obtained, and the search module 14 is matched with the storage module 13 to be used together, so that the distributed characteristic of the distributed storage assembly is further exerted, the log can be searched more efficiently, and the user experience is improved.
In an exemplary embodiment, before the log is stored, the storage module 13 may segment the log, calculate a weight value, store each log and its corresponding segmentation result and weight value together, and when the search module 14 searches the log according to the search condition, rank the logs according to the user search condition according to the segmentation result and weight value corresponding to each log. In word segmentation, text data of a log is split into individual terms, and the terms are standardized. The word segmentation rule can be set in a personalized way according to the needs of users, for example, the word segmentation rule can be divided according to characters such as space, comma and the like, and the word segmentation rule can also be customized based on a regular expression. In addition, the term can be expanded in the word segmentation process, such as synonym expansion, case conversion, stop word filtering and the like, and the original term and the expanded term of the original term can be stored together as a word segmentation result in the storage process, so that the log text can be searched and analyzed more accurately. The importance of the term is mainly considered in weight calculation, and a user can set the weight value of the term according to the actual use requirement so as to highlight the term concerned, so that log information with the term or the synonym term can be arranged in front when searching and sorting, the user can find the required log more quickly, and the working efficiency is improved.
In addition to word segmentation and weight calculation before log storage, when the search module 14 receives a search request and determines a log meeting the conditions, the weight may be calculated according to the search conditions, and the search results may be ranked in a manner not inconsistent with the manner of calculating the weight. For example, for business reasons, a certain architecture needs to pay attention to log information related to deletion operation, and then the distributed storage component performs word segmentation on the log before storing, sets a higher weight value for the log containing the "remove" term, and stores the word segmentation result together with the weight value and the log. If the search condition in the search request received by the search module 14 is { content: "server a" }, which means that the user wants to view the log related to the log content and service a, when the search module 14 ranks the search results, the search module calculates the weight value of "server a" and the weight value of "remove", and the specific scoring function may be configured as required, and ranks the log according to the final weight calculation result.
It should be noted that, the word segmentation rule and the weight rule are preset in the storage module 13, and can be adjusted according to the user requirement under the condition that the user has special requirements.
In an exemplary embodiment, the log service system of the present application further includes an organizing module 15, configured to obtain a log that meets a search condition of a user, organize the obtained log according to a display manner carried in a request sent by the search client, and return the organized log to the search client, so that the search client displays the log according to the display manner. The search request of the search client may include a description of a log display manner, for example, a display item of an information column may be selected, an arrangement order of the information column may be limited, or other display manners required by a user may be limited, etc., the organization module 15 may organize a searched result log according to the display manner in the search request, and return the organized log to the search client, so that the user may see a log search result meeting the display requirement. It should be noted that the organization module included in the flow shown in fig. 2 is an optional module, and a technician can determine whether to use according to the requirement. In the log service system of the embodiment of the application, the user can obtain effective log information more quickly by limiting the display mode of the log, and the user experience is improved.
Based on the log service system provided by the application, the application also provides a log processing method which is applied to the log service system, wherein the log service system comprises a distributed collection component, a self-defined component and a distributed storage component. FIG. 3 is a flowchart of a log processing method according to an exemplary embodiment of the present disclosure, which may include the following steps:
s301: receiving a log sent by the network equipment through the distributed collection component;
s302: the user-defined component filters the collected logs according to the user-defined personalized filtering rules to generate a result log set;
s303: and storing the result log set through the distributed storage component.
As described above, the log processing method further includes:
analyzing the collected logs or the logs contained in the result log set by the user-defined component according to the personalized analysis rules set by the user to obtain the logs in the user-specified format.
As described above, the log service system further includes a distributed message queue, and the log processing method further includes: and sending the collected logs to the custom component through the distributed message queue.
As described above, the log processing method further includes:
storing the collected logs to a disk for backup;
in the event of an anomaly in the system, the collected log is read from disk.
As described above, the log processing method further includes:
and responding to a request sent by a search client, determining a log meeting the search condition in the logs stored by the distributed storage component according to the search condition carried in the request, and returning to the search client.
As described above, the log processing method further includes:
based on a preset word segmentation rule, segmenting each log in the result log set;
calculating a weight value for each log in the result log set based on a preset weight rule;
storing each log and the word segmentation result and the weight value corresponding to each log;
under the condition that a user sends out a search request, sorting the logs meeting the search conditions of the user according to word segmentation results and weight values corresponding to each log.
As described above, the log processing method further includes:
obtaining logs meeting the search conditions of the user, organizing the obtained logs according to the display mode carried in the request sent by the search client, and returning the organized logs to the search client so that the search client displays the logs according to the display mode.
As previously described, the distributed collection component receives logs sent by network devices through multiple pipes that run in separate threads, respectively.
As previously described, the distributed collection component comprises: logstash, flume, filecoat;
the distributed storage component includes: elasticsearch, mysql, oracle;
the distributed message queue includes: kafka, activeMQ, rabbitMQ, rockerMQ.
For the log processing method embodiment, the internal logic of the method embodiment shown in fig. 3 has been described in the system embodiment corresponding to fig. 1, and the specific implementation method thereof may refer to the foregoing embodiment and will not be described herein.
The present application further provides a log processing device, see fig. 4, including:
a collecting unit 41 configured to receive logs sent by the network devices through the distributed collecting component;
a processing unit 42 configured to filter the collected logs by the customization component according to the personalized filtering rules customized by the user, generating a result log set;
a storage unit 43 configured to store the result log set by means of a distributed storage component.
Optionally, the log processing device further includes:
analyzing the collected logs or the logs contained in the result log set by the user-defined component according to the personalized analysis rules set by the user to obtain the logs in the user-specified format.
Optionally, the log processing device further includes: and sending the collected logs to the custom component through the distributed message queue.
Optionally, the log processing device further includes:
the collecting unit 41 is further configured to: storing the collected logs to a disk for backup;
the processing unit 42 is further configured to: in the event of an anomaly in the system, the collected log is read from disk.
Optionally, the log processing device further includes:
and the searching unit 44 is configured to respond to a request sent by a searching client, determine a log meeting the searching condition from the logs stored in the distributed storage component according to the searching condition carried in the request, and return the log to the searching client.
Optionally, the log processing device further includes:
the storage unit 43 is further configured to:
based on a preset word segmentation rule, segmenting each log in the result log set;
calculating a weight value for each log in the result log set based on a preset weight rule;
storing each log and the word segmentation result and the weight value corresponding to each log;
the search unit 44 is further configured to: and sorting the logs meeting the search conditions of the user according to the word segmentation result and the weight value corresponding to each log.
Optionally, the log processing device further includes:
the organizing unit 45 is configured to obtain logs meeting the search conditions of the user, organize the obtained logs according to the display mode carried in the request sent by the search client, and return the organized logs to the search client so that the search client displays according to the display mode.
Optionally, the distributed collection component receives logs sent by the network device through a plurality of pipes, and the plurality of pipes run in independent threads respectively.
Optionally, the distributed collection component includes: logstash, flume, filecoat;
the distributed storage component includes: elasticsearch, mysql, oracle;
the distributed message queue includes: kafka, activeMQ, rabbitMQ, rockerMQ.
The implementation process of the functions and roles of each unit in the above device is specifically shown in the implementation process of the corresponding steps in the above method, and will not be described herein again.
For the device embodiments, reference is made to the description of the method embodiments for the relevant points, since they essentially correspond to the method embodiments. The apparatus embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purposes of the present application. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
Accordingly, the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the log processing method according to any of the foregoing embodiments when executing the computer program.
Referring to fig. 5, at the hardware level, the electronic device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile storage 510, although other services may be required. The processor 502 reads the corresponding computer program from the nonvolatile memory 510 into the memory 408 and then runs, and forms the means of the above log processing method on a logic level. Of course, other implementations, such as logic devices or combinations of hardware and software, are not excluded from the present application, that is, the execution subject of the following processing flows is not limited to each logic unit, but may be hardware or logic devices.
Accordingly, the present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a control method as described in any of the above embodiments.
The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or a combination of any of these devices.
The foregoing is merely a specific implementation of the embodiments of this disclosure, and it should be noted that, for a person skilled in the art, several improvements and modifications may be made without departing from the principles of the embodiments of this disclosure, and these improvements and modifications should also be considered as protective scope of the embodiments of this disclosure.
Claims (10)
1. A log service system, comprising:
the collecting module is used for receiving the logs sent by the network equipment through the distributed collecting components contained in the collecting module;
the processing module is used for acquiring the logs collected by the collecting module, providing the acquired logs to a custom component contained in the processing module, and filtering the acquired logs by the custom component according to personalized filtering rules set by a user to generate a result log set;
and the storage module is used for acquiring the result log set and storing the result log set through a distributed storage component contained in the storage module.
2. The system of claim 1, wherein the processing module is further configured to:
and analyzing the logs collected by the collection module or the logs contained in the result log set by the user-defined component according to the personalized analysis rule set by the user to obtain the logs in the user-specified format.
3. The system of claim 1, wherein the collection module is further configured to: and sending the collected logs to the processing module through a distributed message queue contained in the collecting module.
4. The system of claim 1, wherein the system further comprises a controller configured to control the controller,
the collection module is further configured to: storing the collected logs to a disk for backup;
the processing module is further configured to: and in the case of abnormality of the system, reading the log collected by the collection module from a disk.
5. The system of claim 1, further comprising:
and the search module is used for responding to the request sent by the search client, determining the log meeting the search condition from the logs stored in the storage module according to the search condition carried in the request, and returning to the search client.
6. The system of claim 5, wherein the system further comprises a controller configured to control the controller,
the storage module is further configured to:
based on a preset word segmentation rule, segmenting each log in the result log set;
calculating a weight value for each log in the result log set based on a preset weight rule;
storing each log and the word segmentation result and the weight value corresponding to each log;
the search module is further configured to: and sorting the logs meeting the search conditions of the user according to the word segmentation result and the weight value corresponding to each log.
7. The system of claim 5, further comprising:
the organizing module is used for acquiring logs meeting the search conditions of the user, organizing the acquired logs according to the display mode carried in the request sent by the search client, and returning the organized logs to the search client so that the search client displays the logs according to the display mode.
8. The system of claim 1, wherein the distributed collection component receives logs sent by network devices through a plurality of pipes, each of the plurality of pipes running in a separate thread.
9. The system of claim 1, wherein the system further comprises a controller configured to control the controller,
the distributed collection assembly includes: logstash, flume, filecoat;
the distributed storage component includes: elasticsearch, mysql, oracle;
the distributed message queue includes: kafka, activeMQ, rabbitMQ, rockerMQ.
10. The log processing method is characterized by being applied to a log service system, wherein the log service system comprises a distributed collection component, a custom component and a distributed storage component, and the method comprises the following steps:
receiving a log sent by the network equipment through the distributed collection component;
the user-defined component filters the collected logs according to the user-defined personalized filtering rules to generate a result log set;
and storing the result log set through the distributed storage component.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310415428.1A CN116489005A (en) | 2023-04-12 | 2023-04-12 | Log service system and log processing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310415428.1A CN116489005A (en) | 2023-04-12 | 2023-04-12 | Log service system and log processing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116489005A true CN116489005A (en) | 2023-07-25 |
Family
ID=87213209
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310415428.1A Pending CN116489005A (en) | 2023-04-12 | 2023-04-12 | Log service system and log processing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116489005A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117033464A (en) * | 2023-08-11 | 2023-11-10 | 上海鼎茂信息技术有限公司 | Log parallel analysis algorithm based on clustering and application |
-
2023
- 2023-04-12 CN CN202310415428.1A patent/CN116489005A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117033464A (en) * | 2023-08-11 | 2023-11-10 | 上海鼎茂信息技术有限公司 | Log parallel analysis algorithm based on clustering and application |
| CN117033464B (en) * | 2023-08-11 | 2024-04-02 | 上海鼎茂信息技术有限公司 | Log parallel analysis algorithm based on clustering and application |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11677635B2 (en) | Hierarchical network analysis service | |
| US11513935B2 (en) | System and method for detecting anomalies by discovering sequences in log entries | |
| EP4099170B1 (en) | Method and apparatus of auditing log, electronic device, and medium | |
| CN112416724B (en) | Alarm processing method, system, computer device and storage medium | |
| CN110309030A (en) | Log analysis monitoring system and method based on ELK and Zabbix | |
| CN111881011A (en) | Log management method, platform, server and storage medium | |
| WO2003073203A2 (en) | System and method for analyzing input/output activity on local attached storage | |
| JP4506520B2 (en) | Management server, message extraction method, and program | |
| CN111046011A (en) | Log collection method, system, node, electronic device and readable storage medium | |
| CN110912757B (en) | Service monitoring method and server | |
| KR102580916B1 (en) | Apparatus and method for managing trouble using big data of 5G distributed cloud system | |
| US20230047615A1 (en) | Communication Device, Surveillance Server, and Log Collection Method | |
| US8909768B1 (en) | Monitoring of metrics to identify abnormalities in a large scale distributed computing environment | |
| CN116489005A (en) | Log service system and log processing method | |
| CN108984362A (en) | Log collection method and device, storage medium, electronic equipment | |
| US11822578B2 (en) | Matching machine generated data entries to pattern clusters | |
| CN114143169A (en) | Micro-service application observability system | |
| CN113760856A (en) | Database management method and device, computer readable storage medium and electronic device | |
| CN111723262A (en) | System and method for combining headlines, abstracts and texts of network news in batches | |
| CN111290909A (en) | System and method for monitoring and alarming ceph cluster | |
| CN117591381B (en) | Data reporting method and device | |
| US20240028237A1 (en) | Method and system for health driven network slicing based data migration | |
| CN118210682B (en) | Multi-dimensional monitoring and obstacle removing method and system for distributed architecture system | |
| CN113900898B (en) | Data processing system, equipment and medium | |
| CN116431872B (en) | Observable system and service observing method based on observable system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |