CN109995589B - Log collection method and system - Google Patents

Log collection method and system Download PDF

Info

Publication number
CN109995589B
CN109995589B CN201910255934.2A CN201910255934A CN109995589B CN 109995589 B CN109995589 B CN 109995589B CN 201910255934 A CN201910255934 A CN 201910255934A CN 109995589 B CN109995589 B CN 109995589B
Authority
CN
China
Prior art keywords
log
message
sending node
receiving end
node
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910255934.2A
Other languages
Chinese (zh)
Other versions
CN109995589A (en
Inventor
雷克
朱磊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
New H3C Big Data Technologies Co Ltd
Original Assignee
New H3C Big Data Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by New H3C Big Data Technologies Co Ltd filed Critical New H3C Big Data Technologies Co Ltd
Priority to CN201910255934.2A priority Critical patent/CN109995589B/en
Publication of CN109995589A publication Critical patent/CN109995589A/en
Application granted granted Critical
Publication of CN109995589B publication Critical patent/CN109995589B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06Management of faults, events, alarms or notifications
    • H04L41/069Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/56Provisioning of proxy services
    • H04L67/565Conversion or adaptation of application format or content

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The disclosure provides a log collection method and a log collection system, and relates to the technical field of communication. The method comprises the following steps: a log receiving end monitors a preset port to obtain a log from at least one log sending node; for each acquired log, the log receiving end converts the log into a first message and stores the first message in a first message queue; and the distribution node determines the identifier of the log sending node corresponding to any first message acquired from the first message queue, converts the first message into a second message and stores the second message in a second message queue corresponding to the identifier of the log sending node. Through monitoring the preset ports, other ports of the log receiving end do not need to be monitored, a plurality of tasks do not need to be started to monitor a plurality of ports, resources occupied by the log receiving end are reduced, the utilization rate of the log receiving end is improved, and the flexibility of obtaining logs is improved.

Description

Log collection method and system
Technical Field
The present disclosure relates to the field of communications technologies, and in particular, to a log collection method and system.
Background
With the continuous development of internet technology, more and more log data are generated by network equipment, and the log data can be collected and analyzed to obtain valuable contents in the log data according to the analysis result.
In the related art, a log receiving end can start a plurality of logstack (data collection engine) tasks, monitor a plurality of ports, receive log data with different sources and coding formats by adopting different configuration files, and finally send the received log data to different message queues.
However, the log receiving end needs to start multiple logstack tasks to monitor each port, which causes a problem that starting multiple logstack tasks occupies more resources.
Disclosure of Invention
An object of the present disclosure is to provide a log collection method, to solve the problem that a log receiving end needs to occupy too many resources when collecting log data, in order to overcome the above-mentioned shortcomings in the prior art.
In order to achieve the above purpose, the embodiments of the present disclosure adopt the following technical solutions:
in a first aspect, an embodiment of the present disclosure provides a log collecting method, where the method is applied to a log collecting system, where the log collecting system includes at least one log sending node, a log receiving end, and a distribution node, and the method includes:
the log receiving end monitors a preset port to obtain a log from at least one log sending node;
for each acquired log, the log receiving terminal converts the log into a first message and stores the first message in a first message queue;
and the distribution node determines the identifier of the log sending node corresponding to any first message acquired from the first message queue, converts the first message into a second message and stores the second message in a second message queue corresponding to the identifier of the log sending node.
Optionally, each log sending node is configured with unified information of a log receiving end, where the information of the log receiving end includes: and the address of the log receiving end and the preset port number of the port.
Optionally, for each log obtained, the first message includes: the log and an identifier of a log sending node sending the log.
Optionally, the determining, by the distribution node, an identifier of a log sending node corresponding to any first message obtained from the first message queue, where the determining includes:
and the distribution node decodes any first message acquired from the first message queue by adopting a preset first coding format, and acquires the identifier of the log sending node contained in the first message.
Optionally, the converting the first message into a second message includes:
the distribution node searches a configuration file matched with the acquired identification of the log sending node in a pre-configured configuration file of the log sending node;
and the distribution node encodes the content of the first message decoded by adopting the first encoding format according to the second encoding format recorded in the matched configuration file to obtain the second message.
In a second aspect, an embodiment of the present disclosure further provides a log collecting system, where the log collecting system includes: the system comprises at least one log sending node, a log receiving end and a distributing node;
the log receiving end is used for monitoring a preset port and acquiring a log from the at least one log sending node;
for each acquired log, the log receiving end is further used for converting the log into a first message and storing the first message in a first message queue;
the distribution node is used for determining the identifier of the log sending node corresponding to any first message acquired from the first message queue, converting the first message into a second message and storing the second message in a second message queue corresponding to the identifier of the log sending node.
Optionally, each log sending node is configured with unified information of a log receiving end, where the information of the log receiving end includes: and the address of the log receiving end and the preset port number of the port.
Optionally, for each log obtained, the first message includes: the log and an identifier of a log sending node sending the log.
Optionally, the distribution node is specifically configured to decode any first message obtained from the first message queue by using a preset first coding format, and obtain an identifier of the log sending node included in the first message.
Optionally, the distribution node is specifically configured to search, in a configuration file of a pre-configured log sending node, a configuration file matched with the obtained identifier of the log sending node;
the distribution node is further specifically configured to encode, according to the second encoding format recorded in the matched configuration file, the content of the first message decoded by using the first encoding format, to obtain the second message.
The beneficial effect of this disclosure is:
the log sending method and the log sending device have the advantages that the preset port is monitored through the log receiving end, the log from at least one log sending node is obtained, for each obtained log, the log receiving end converts the log into a first message and stores the first message in the first message queue, the distribution node determines the identification of the log sending node corresponding to the first message for any first message obtained from the first message queue, converts the first message into a second message and stores the second message in the second message queue corresponding to the identification of the log sending node. The log collection can be completed only by monitoring the preset port without monitoring other ports of a log receiving terminal, and the log collection can be completed without starting a plurality of tasks to monitor the plurality of ports, so that resources occupied by the log receiving terminal are reduced, the utilization rate of the log receiving terminal is improved, and the flexibility of log acquisition is improved.
Drawings
To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings that are required to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present disclosure and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings may be obtained from the drawings without inventive effort.
Fig. 1 is a schematic structural diagram of a log collection system related to a log collection method provided by the present disclosure;
fig. 2 is a schematic diagram of a log collection process provided by the present disclosure;
fig. 3 is a schematic flowchart of a log collection method according to an embodiment of the present disclosure;
fig. 4 is a schematic diagram of a log collection system provided in an embodiment of the present disclosure;
fig. 5 is a schematic diagram of a log collection device according to an embodiment of the present disclosure.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some, but not all embodiments of the present disclosure.
Fig. 1 is a schematic structural diagram of a log collection system related to a log collection method provided by the present disclosure; as shown in fig. 1, the log collection system may include: a log sender 110, a log receiver 120, a first message queue 130, a distribution node 140, and a second message queue 150.
Multiple log sender nodes (also referred to as services) may be deployed at log sender 110, and these services may be deployed on one or more devices. In the present disclosure, unified log receiver 120 information may be configured at the log sender 110, where the information includes an IP (Internet Protocol) address, a port, and the like, that is, the log receiver information configured on each log sender is the same, and based on this, when a new log sender is added, the log receiver 120 information is configured for the log sender, since the log receivers configured on all log senders have the same information and cannot be changed by adding or reducing the log senders, a restart operation is not required to be performed on the log receiver 120 (that is, the log receiver port information does not need to be updated). Each log sending node can send the log to the log receiving end 120 according to the configured information of the log receiving end 120.
Only one log receiving node (also referred to as a service) may be deployed at the log receiving end 120, or multiple log receiving nodes may be deployed for load balancing, and the multiple log receiving nodes use a uniform IP address and port, where the IP address and port are the IP address and port configured at the log sending end 110. The Log receiving service in the present disclosure may be RSYSLOG (socket-fast System for Log). Compared with the method for monitoring the receiving port by starting the Logstash, the method for monitoring the receiving port by starting the RSYSLOG can process log data with larger data volume, and avoids the problem of log data loss caused by larger data volume of the log data.
The log receiving end 120 processes the received logs from different log sending nodes and sends the processed logs to a unified message queue, i.e. a first message queue.
The distribution node 140, i.e., a distribution service (in the example of the present disclosure, one distribution service is deployed for example), may consume the logs from different log sending nodes saved in the first message queue in a uniform encoding format. The distribution service may be java distribution service. The distribution node is pre-configured with a configuration file corresponding to each log transmission node. According to the configuration file, the distribution node can encode the log according to the encoding format corresponding to the log and then send the log to the corresponding second message queue, namely, a plurality of second message queues exist.
As shown in fig. 2, the log collection process provided by the present disclosure includes:
1. each log sending node of the log sending end is configured with information of the log receiving end, and the information is assumed to be an IP address: 10.121.37.15, port number: 514;
taking the log sending node 1 as an example: a log sending node 1 (with an IP address of 192.168.0.1) sends a TCP (Transmission Control Protocol) or UDP (User Datagram Protocol) message (carrying a log 1) to a log receiving end, where the content of the log 1 is: "hello world! ".
2. The log receiving end monitors a port with a port number of 514, acquires a TCP or UDP packet, acquires log 1 after parsing, converts the log 1 into a message 1 in a JSON (JavaScript Object Notation) format, and sends the message 1 to a first message queue, which may be a kafka queue in this example. The message 1 contains the following contents: original Log 1 content "hello world! "and the source IP of log 1, i.e. the IP address 192.168.0.1 of the log-sending node 1.
3. The distribution node consumes (decodes) the messages in the first message queue using the ISO (International Organization for Standardization) -8859-1 encoding format in unison. After the message 1 is decoded by adopting an ISO-8859-1 encoding format, the source IP of the log 1 in the message can be acquired: 192.168.0.1.
the distribution node is pre-configured with a configuration file corresponding to each log sending node, wherein the configuration file at least comprises an IP address and an encoding format of the log sending node; therefore, the source IP obtained by decoding the message 1 can be matched with the configuration file of the log sending node 1 configured in advance on the distribution node, and the content of the message 1 is re-encoded according to the encoding format recorded in the configuration file to obtain the message 2.
And the distribution node sends the message 2 to a corresponding second message queue according to the source IP of the log 1. Wherein, the second message queue can also be a kafka queue.
Fig. 3 is a schematic flowchart of a log collecting method according to an embodiment of the present disclosure, which is applied to the log collecting system shown in fig. 1, and as shown in fig. 3, the method includes:
step 301, the log receiving end monitors a preset port to obtain a log from at least one log sending node.
Each log sending node is configured with unified information of a log receiving end, where the information of the log receiving end may include: the address of the log receiving end and the preset port number of the port.
In order to reduce the resources occupied by the log receiving end when collecting the logs, the log receiving end can receive the logs sent by the plurality of log sending nodes through only one port in a mode of configuring the port.
Therefore, the log receiving end can receive different logs sent by different log sending nodes only by monitoring the configured ports, and does not need to monitor a plurality of ports, so that resources occupied by the log receiving end are reduced.
Specifically, after configuring the port of the log receiving end, the log receiving end may monitor the configured port, and detect whether the log sending end sends the message including the log to the configured port of the log receiving end.
Correspondingly, if the log receiving end detects that the preset port receives the message which is sent by at least one log sending node and comprises the log, the message can be received through the port, and the message is analyzed to obtain the log sent by the log sending node, so that in the subsequent steps, the obtained log can be analyzed, and the log obtained by analysis is recoded and stored in the preset message queue.
For example, if the information of the log receiving end stored by the log sending end includes: IP address of log receiving end: 10.121.37.15 and port number of the preconfigured port: 514, the first sending node of the log sending end may send the log content of "hello world |" to the log receiving end corresponding to the information of the log receiving end by using a TCP message or UDP message! "is recorded.
Correspondingly, the log receiving end can monitor the port with the port number 514, when the message sent by the first sending node is detected, the message including the log can be received through the port with the port number 514, and the message is analyzed to obtain the log content' hello world! "is recorded.
Step 302, for each obtained log, the log receiving end converts the log into a first message and stores the first message in a first message queue.
Wherein, for each log obtained, the first message may include: the log and an identification of the log sending node that sent the log.
After the log receiving end obtains the log sent by the log sending end, the log receiving end can preprocess the received log to obtain first messages, so that the first messages are cached in a first message queue, and in the subsequent step, each first message can be distributed to different second message queues according to the identification corresponding to each first message in the first message queue.
Further, in order to maintain the integrity of the log and avoid the information indicated by the log from being distorted, the received log may be preprocessed in an encapsulating manner. In addition, because the logs are received through the same receiving port, in practical application, the logs from different sources need to be processed differently, so in the process of encapsulating and caching the logs, the source IPs of the logs can be encapsulated together, and the source IPs are the IP addresses corresponding to the log sending nodes sending the logs.
Therefore, the identifier of the log sending node that sends the log in the first message may be an IP of a source of the log, and of course, the identifier may also be other information indicating the log sending node, which is not limited in this disclosure.
Specifically, the log receiving end may first obtain an address of a log sending node that sends each log, obtain a source IP corresponding to each log, and package the source IP as an information field together with the corresponding log according to a preset package format to obtain first messages corresponding to each log, so as to store each first message obtained by conversion in the first message queue.
For example, corresponding to the example in step 301, if the IP address corresponding to the first sending node is 192.168.0.1, the log receiving end may first obtain the IP address as the source IP of the log, and according to the JSON format, perform the log processing on the source IP "192.168.0.1" and the log content "hello world |)! "encapsulate together, get the encapsulated first message 1, and encapsulate this first message 1 in the first message queue in the form of kafka message queue.
Step 303, the distributing node determines, for any first message obtained from the first message queue, an identifier of a log sending node corresponding to the first message, converts the first message into a second message, and stores the second message in a second message queue corresponding to the identifier of the log sending node.
After buffering the first message through the first message queue, the encapsulated first message may be distributed to a different second message queue for storage. However, because the logs corresponding to the first messages are all received through the same port of the log receiving end, the first messages cannot be distributed according to the port for receiving the logs. Therefore, it is necessary to obtain the identifier of the log sending node included in each first message, that is, to send the address information of the log sending node corresponding to the log in each first message, so as to send the encapsulated first message to a different second message queue according to the identifier of the log sending node corresponding to each first message.
Therefore, the distribution node may decode any first message obtained from the first message queue by using a preset first encoding format, and obtain an identifier of the log sending node included in the first message.
For example, corresponding to the example of step 302, the first encoding format may be an ISO-8859-1 encoding format, and the distribution node may decode the first message 1 in the first message queue using the ISO-8859-1 encoding format to obtain the log "hello world!in the first message 1! "and the identifier corresponding to the first sending node sending the log, that is, the IP address 192.168.0.1 of the first sending node.
Further, in order to convert the first message into a second message that can be recognized by a different device, a pre-stored configuration file corresponding to the identifier may be selected from the distribution node according to the identifier of the log sending node included in the first message, so that the first message is encoded again according to the encoding format indicated by the configuration file, and the second message is obtained.
Optionally, in the process of converting the first message into the second message, the distribution node may search, in the configuration file of the pre-configured log sending node, a configuration file matched with the obtained identifier of the log sending node, and encode, according to the second encoding format recorded in the matched configuration file, the content of the first message decoded by using the first encoding format, to obtain the second message.
Specifically, the distribution node may select, according to the obtained identifier of the log sending node, a configuration file matched with the identifier from a plurality of pre-stored configuration files, so as to encode the first message decoded by using the first encoding format again according to the second encoding format indicated by the configuration file, thereby obtaining a second message that can be recognized by the corresponding device, and store the second message in a second message queue corresponding to the identifier according to the obtained identifier of the log sending node. It should be noted that, the distribution node stores in advance a configuration file corresponding to each log sending node, where the configuration file includes an identifier of the log sending node, that is, an IP address of the log sending node, and a second encoding format for encoding the log sent by the log sending node.
For example, if the distribution node stores a plurality of configuration files corresponding to different log sending nodes, after decoding the first message 1, the distribution node obtains an identifier of the first sending node, that is, the IP address 192.168.0.1 of the first sending node, and the distribution node may match the IP address 192.168.0.1 with the IP address corresponding to each configuration file to obtain the configuration file corresponding to the first sending node, so as to encode the decoded first message 1 again according to the second encoding format recorded in the configuration file to obtain the second message 1.
Further, the distribution node may also store the obtained second message 1 in the second message queue corresponding to the IP address according to the IP address 192.168.0.1. The second message queue may also be a kafka queue, which is not limited in this disclosure.
It should be noted that, in practical applications, the master node of the log receiving end may be abnormal, which results in that log data cannot be collected continuously, so that collection may be performed in a load balancing manner by using a LVS (Linux Virtual Server). For example, when the master node at the log receiving end is abnormal, the backup node can continue to collect log data, so as to prevent data loss caused by the abnormal master node.
To sum up, the log collecting method provided by the embodiment of the present disclosure obtains logs from at least one log sending node by monitoring a preset port through a log receiving end, and for each obtained log, the log receiving end converts the log into a first message and stores the first message in a first message queue, and the distribution node determines, by using a node pair, an identifier of a log sending node corresponding to the first message, and converts the first message into a second message and stores the second message in a second message queue corresponding to the identifier of the log sending node. The log collection can be completed only by monitoring the preset port without monitoring other ports of a log receiving terminal, and the log collection can be completed without starting a plurality of tasks to monitor the plurality of ports, so that resources occupied by the log receiving terminal are reduced, the utilization rate of the log receiving terminal is improved, and the flexibility of log acquisition is improved.
Fig. 4 is a schematic diagram of a log collection system according to an embodiment of the present disclosure, and as shown in fig. 4, the log collection system specifically includes: at least one log sending node 401, a log receiving end 402 and a distributing node 403;
the log receiving end 402 is configured to monitor a preset port and obtain a log from the at least one log sending node 401;
for each acquired log, the log receiving end 402 is further configured to convert the log into a first message and store the first message in a first message queue;
the distribution node 403 is configured to determine, for any first message acquired from the first message queue, an identifier of the log sending node 401 corresponding to the first message, convert the first message into a second message, and store the second message in a second message queue corresponding to the identifier of the log sending node 401.
Optionally, each log sending node 401 is configured with unified information of the log receiving end 402, where the information of the log receiving end 402 includes: the address of the log receiving terminal 402 and the port number of the preset port.
Optionally, for each log obtained, the first message includes: the log and the identity of the log sending node 401 that sent the log.
Optionally, the distribution node 403 is specifically configured to decode any first message obtained from the first message queue by using a preset first encoding format, and obtain an identifier of the log sending node 401 included in the first message.
Optionally, the distribution node 403 is specifically configured to search, in a configuration file of the log sending node 401 configured in advance, a configuration file matched with the acquired identifier of the log sending node 401;
the distribution node 403 is further specifically configured to encode the content of the first message decoded by using the first encoding format according to the second encoding format recorded in the matched configuration file, so as to obtain a second message.
To sum up, the log collecting system provided in the embodiment of the present disclosure obtains logs from at least one log sending node by monitoring a preset port through a log receiving end, and for each obtained log, the log receiving end converts the log into a first message and stores the first message in a first message queue, and a distribution node determines an identifier of a log sending node corresponding to the first message for any first message obtained from the first message queue, converts the first message into a second message, and stores the second message in a second message queue corresponding to the identifier of the log sending node. The log collection can be completed only by monitoring the preset port without monitoring other ports of a log receiving terminal, and the log collection can be completed without starting a plurality of tasks to monitor the plurality of ports, so that resources occupied by the log receiving terminal are reduced, the utilization rate of the log receiving terminal is improved, and the flexibility of log acquisition is improved.
The system is used for executing the method provided by the foregoing embodiment, and the implementation principle and technical effect are similar, which are not described herein again.
Fig. 5 is a schematic diagram of a log collecting apparatus according to an embodiment of the present disclosure, where the apparatus may be integrated in a terminal device or a chip of the terminal device, and the terminal may be a computing device with a log collecting function.
The device comprises: memory 501, processor 502.
The memory 501 is used for storing programs, and the processor 502 calls the programs stored in the memory 501 to execute the above-mentioned method embodiments. The specific implementation and technical effects are similar, and are not described herein again.
Optionally, the present disclosure also provides a program product, e.g. a computer readable storage medium, comprising a program which, when executed by a processor, is adapted to perform the above-described method embodiments.
In the several embodiments provided in the present disclosure, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily think of the changes or substitutions within the technical scope of the present application, and shall cover the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (4)

1. A log collection method is applied to a log collection system, the log collection system comprises at least one log sending node, a log receiving end and a distribution node, and the method comprises the following steps:
the log receiving end monitors a preset port to obtain a log from at least one log sending node;
for each acquired log, the log receiving terminal converts the log into a first message and stores the first message in a first message queue;
the distribution node determines the identifier of a log sending node corresponding to any first message acquired from the first message queue, converts the first message into a second message and stores the second message in a second message queue corresponding to the identifier of the log sending node;
each log sending node is configured with unified information of a log receiving end, wherein the information of the log receiving end comprises: the address of the log receiving end and the port number of the preset port;
the determining, by the distribution node, an identifier of a log sending node corresponding to any first message acquired from the first message queue includes:
the distribution node decodes any first message acquired from the first message queue by adopting a preset first coding format, and acquires the identifier of the log sending node contained in the first message;
the converting the first message into a second message comprises:
the distribution node searches a configuration file matched with the acquired identification of the log sending node in a pre-configured configuration file of the log sending node;
and the distribution node encodes the content of the first message decoded by adopting the first encoding format according to the second encoding format recorded in the matched configuration file to obtain the second message.
2. The log collection method of claim 1, wherein for each log obtained, the first message comprises: and the log and the identifier of the log sending node sending the log.
3. A log collection system, comprising: the system comprises at least one log sending node, a log receiving end and a distributing node;
the log receiving end is used for monitoring a preset port and acquiring logs from the at least one log sending node;
for each acquired log, the log receiving end is further used for converting the log into a first message and storing the first message in a first message queue;
the distribution node is used for determining the identifier of the log sending node corresponding to any first message acquired from the first message queue, converting the first message into a second message and storing the second message into a second message queue corresponding to the identifier of the log sending node;
each log sending node is configured with unified information of a log receiving end, wherein the information of the log receiving end comprises: the address of the log receiving end and the port number of the preset port;
the distribution node is specifically configured to decode any first message obtained from the first message queue by using a preset first coding format, and obtain an identifier of the log sending node included in the first message;
the distribution node is specifically configured to search a configuration file matched with the acquired identifier of the log sending node from configuration files of pre-configured log sending nodes;
the distribution node is further specifically configured to encode, according to the second encoding format recorded in the matched configuration file, the content of the first message decoded by using the first encoding format, to obtain the second message.
4. The log collection system of claim 3, wherein the first message comprises, for each log retrieved: and the log and the identifier of the log sending node sending the log.
CN201910255934.2A 2019-03-29 2019-03-29 Log collection method and system Active CN109995589B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910255934.2A CN109995589B (en) 2019-03-29 2019-03-29 Log collection method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910255934.2A CN109995589B (en) 2019-03-29 2019-03-29 Log collection method and system

Publications (2)

Publication Number Publication Date
CN109995589A CN109995589A (en) 2019-07-09
CN109995589B true CN109995589B (en) 2022-09-09

Family

ID=67131879

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910255934.2A Active CN109995589B (en) 2019-03-29 2019-03-29 Log collection method and system

Country Status (1)

Country Link
CN (1) CN109995589B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111221664A (en) * 2019-12-24 2020-06-02 视联动力信息技术股份有限公司 Operation log processing system

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2419305C (en) * 2003-02-20 2006-03-21 Ibm Canada Limited - Ibm Canada Limitee Unified logging service for distributed applications
CN100521623C (en) * 2007-05-22 2009-07-29 网御神州科技(北京)有限公司 High-performance Syslog processing and storage method
US9917800B2 (en) * 2012-08-24 2018-03-13 Cisco Technology, Inc. System and method for centralized virtual interface card driver logging in a network environment

Also Published As

Publication number Publication date
CN109995589A (en) 2019-07-09

Similar Documents

Publication Publication Date Title
CN109391500B (en) Configuration management method, device and equipment
AU2002240200B2 (en) A digital television application protocol for interactive television
CN102655481B (en) A kind of instant messaging chat content inspection method based on webpage and system
CN108009028B (en) Message processing method, device, equipment and computer readable storage medium
AU2002240200A1 (en) A digital television application protocol for interactive television
CN105897888B (en) Client connection method and system
CN111294235A (en) Data processing method, device, gateway and readable storage medium
CN111917900A (en) Request processing method and device for domain name proxy
CN112532673A (en) Message sending method and device, computer readable storage medium and electronic equipment
US20150067066A1 (en) Provisioning Communication Services using Proxy Server in a Cloud
CN109995589B (en) Log collection method and system
CN104052679A (en) Load balancing method and device for network flow
CN115039392A (en) Message processing method, device and system and server
CN109788251B (en) Video processing method, device and storage medium
CN108924773B (en) Message processing method and device
CN111740967A (en) Method, system and electronic equipment for message conversion and transmission
CN109766347B (en) Data updating method, device, system, computer equipment and storage medium
CN114531332A (en) Data transmission method, data processing method, device and storage medium
CN112737947B (en) Virtual network cross-domain transmission method, system, equipment and medium based on MPLS
CN115695594B (en) Internet of things data communication method and device
CN115802098B (en) Cloud application data interaction method, client, rendering end and system
WO2024061189A1 (en) Service mesh system and information transmission method based on service mesh system
CN115118765B (en) Service processing method, device, electronic equipment and storage medium
CN112769741B (en) Message communication method and electronic equipment
CN117692399A (en) Message uploading method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant