CN111737207A - Method and device for displaying and collecting logs of service nodes in distributed system - Google Patents

Method and device for displaying and collecting logs of service nodes in distributed system Download PDF

Info

Publication number
CN111737207A
CN111737207A CN202010570955.6A CN202010570955A CN111737207A CN 111737207 A CN111737207 A CN 111737207A CN 202010570955 A CN202010570955 A CN 202010570955A CN 111737207 A CN111737207 A CN 111737207A
Authority
CN
China
Prior art keywords
log
service
log file
file
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010570955.6A
Other languages
Chinese (zh)
Other versions
CN111737207B (en
Inventor
赵宇
侯雪峰
徐寅斐
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Kingsoft Cloud Network Technology Co Ltd
Original Assignee
Beijing Kingsoft Cloud Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Kingsoft Cloud Network Technology Co Ltd filed Critical Beijing Kingsoft Cloud Network Technology Co Ltd
Priority to CN202010570955.6A priority Critical patent/CN111737207B/en
Publication of CN111737207A publication Critical patent/CN111737207A/en
Application granted granted Critical
Publication of CN111737207B publication Critical patent/CN111737207B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/16File or folder operations, e.g. details of user interfaces specifically adapted to file systems
    • G06F16/162Delete operations
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/1805Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815Journaling file systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10File systems; File servers
    • G06F16/18File system types
    • G06F16/182Distributed file systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The embodiment of the invention provides a method and a device for displaying and collecting logs of service nodes in a distributed system, which relate to the technical field of Internet, wherein a log display server receives log files sent by each service node; associating the log file with a corresponding service program according to the log information included in the log file; and when the log content viewing request is acquired, displaying a service program name list operated in the target service node, a log file name list of the target service program and target log content according to the incidence relation between the log file and the service program. In the method, the log display server is respectively in communication connection with the plurality of service nodes in the distributed system, the log file can be displayed to the log display server in real time through the deployment method without complex components and with light weight, the deployment and maintenance cost is low, and meanwhile, the service operation and maintenance efficiency of the distributed system is improved.

Description

Method and device for displaying and collecting logs of service nodes in distributed system
Technical Field
The invention relates to the technical field of internet, in particular to a method and a device for displaying and collecting logs of service nodes in a distributed system.
Background
With the development of internet technology, the system service architecture is also rapidly developing and changing. In consideration of poor loadability and disaster tolerance of a stand-alone system, the current system design usually adopts a distributed system, that is, a multi-machine deployment mode is used to support the same service together, such as common large data component Hadoop, Hbase, Elasticsearch and other services.
Meanwhile, the problem brought by multi-machine deployment is that operation and maintenance are complex, a common means for service operation and maintenance is to check logs, and the check logs generally need to be logged in corresponding machine nodes for checking, so that the problem of low service operation and maintenance efficiency is caused.
Disclosure of Invention
The invention aims to provide a method and a device for displaying and collecting logs of service nodes in a distributed system, which are used for displaying log files in real time, reducing deployment and maintenance cost and improving service operation and maintenance efficiency of the distributed system.
In a first aspect, an embodiment of the present invention provides a method for displaying a log of a service node in a distributed system, where the distributed system includes a plurality of service nodes, each service node runs at least one service program, the method is applied to a log display server, and the log display server is respectively in communication connection with the plurality of service nodes in the distributed system, and the method includes: receiving a log file sent by each service node, wherein the log file is an operation log of a corresponding service program in the corresponding service node; the log file comprises log information and log content, wherein the log information comprises a service node name, a service program name and a log file name; associating the log file with the corresponding service program according to the log information; when a log content viewing request of a user for a target service program in a target service node is acquired, a service program name list, a log file name list of the target service program and target log content which are operated in the target service node are displayed according to the incidence relation between the log file and the service program.
Further, associating the log file with the corresponding service program according to the log information includes: analyzing the log information to obtain a service node name, a service program name and a log file name; according to the service node name, the log file is associated to a service node level list in a preset directory; according to the service program name, the log file is associated to a service program level list in a preset directory; and according to the name of the log file, associating the log file to a log file hierarchy list in a preset directory.
In a second aspect, an embodiment of the present invention provides a method for aggregating logs of service nodes in a distributed system, where the distributed system includes a plurality of service nodes, each service node runs at least one service program, and each service node is in communication connection with a log display server, and the method is applied to the service nodes, and includes: every interval of a first preset period, acquiring a log file of a specified target service program in a configuration file according to the preset configuration file; the log file is an operation log of a corresponding service program in a corresponding service node; the log file comprises log information and log content, wherein the log information comprises a service node name, a service program name and a log file name; and transmitting the log file to a log display server.
Further, the step of obtaining the log file of the target service program specified in the configuration file includes: determining a first target log file according to a target service program specified by the configuration file; acquiring the number of newly added log records in a first target log file; and acquiring the log records with the same number as the number of the pieces from the tail part of the first target log file according to the number of the pieces, and sending the log records as log files.
Further, the step of obtaining the log file of the target service program specified in the configuration file includes: and determining a second target log file to be sent as a log file according to the target service program specified by the configuration file, wherein the second target log file is in a compressed packet format.
Further, the step of obtaining the log file of the target service program specified in the configuration file includes: determining the port number of the target service program according to the configuration file; monitoring the port number, acquiring a resource use log of the target service program to the service node, and sending the resource use log as a log file.
Further, after the step of transmitting the log file to the log display server, the method further includes: and if the transmission of the log file fails, saving the log file to a local temporary directory, and returning to the step of transmitting the log file to the log display server after a second preset period.
Further, after the step of transmitting the log file to the log display server, the method further includes: and if the log file is successfully transmitted, deleting the log file in the temporary directory.
In a third aspect, an embodiment of the present invention provides an apparatus for displaying a log of a service node in a distributed system, where the distributed system includes a plurality of service nodes, each service node runs at least one service program, the apparatus is disposed at a log display server, and the log display server is respectively in communication connection with the plurality of service nodes in the distributed system, and the apparatus includes: the receiving module is used for receiving the log files sent by each service node, and the log files are the running logs of the corresponding service programs in the corresponding service nodes; the log file comprises log information and log content, wherein the log information comprises a service node name, a service program name and a log file name; the correlation module is used for correlating the log file with the corresponding service program according to the log information; and the display module is used for displaying a service program name list operated in the target service node, a log file name list of the target service program and target log content according to the incidence relation between the log file and the service program when a log content viewing request of a user for the target service program in the target service node is obtained.
In a fourth aspect, an embodiment of the present invention provides an apparatus for collecting logs of service nodes in a distributed system, where the distributed system includes a plurality of service nodes, each service node runs at least one service program, each service node is in communication connection with a log display server, and the apparatus is disposed at a service node, and includes: the acquisition module is used for acquiring a log file of a specified target service program in a configuration file according to the preset configuration file every a first preset period; the log file is an operation log of a corresponding service program in a corresponding service node; the log file comprises log information and log content, wherein the log information comprises a service node name, a service program name and a log file name; and the transmission module is used for transmitting the log file to the log display server.
In a fifth aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, where the memory stores a computer program operable on the processor, and the processor implements, when executing the computer program, the method for presenting a log of a service node in a distributed system according to any one of the first aspect or the method for aggregating logs of service nodes in a distributed system according to any one of the second aspect.
In a sixth aspect, embodiments of the present invention provide a computer readable storage medium storing machine executable instructions which, when invoked and executed by a processor, cause the processor to perform the steps of the method of any one of the first aspect for exposing a log of a service node in a distributed system, or the method of any one of the second aspect for aggregating logs of service nodes in a distributed system.
The method and the device for displaying and collecting the logs of the service nodes in the distributed system are provided by the embodiment of the invention, wherein the method for displaying the logs of the service nodes in the distributed system is applied to a log display server and used for receiving log files sent by all the service nodes; associating the log file with a corresponding service program according to the log information included in the log file; and when the log content viewing request is acquired, displaying a service program name list operated in the target service node, a log file name list of the target service program and target log content according to the incidence relation between the log file and the service program. In the method, the distributed system comprises a plurality of service nodes, at least one service program runs in each service node, the log display service end is in communication connection with the service nodes in the distributed system respectively, the log file can be displayed to the log display service end in real time through the deployment method without complex components and with light weight, the deployment and maintenance cost is low, and meanwhile, the service operation and maintenance efficiency of the distributed system is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic diagram of a distributed system according to an embodiment of the present invention;
fig. 2 is a flowchart of a method for displaying a log of a service node in a distributed system according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a directory of a log display server according to an embodiment of the present invention;
fig. 4 is a flowchart of a method for aggregating logs of service nodes in a distributed system according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an apparatus for displaying a log of a service node in a distributed system according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an apparatus for aggregating logs of service nodes in a distributed system according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terms "comprising" and "having," and any variations thereof, as referred to in embodiments of the present invention, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements but may alternatively include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
With the development of internet technology, the system service architecture is also rapidly developing and changing. In consideration of poor loadability and disaster tolerance of a stand-alone system, a distributed system is usually adopted in the current system design, i.e., a multi-machine deployment mode is used for jointly supporting the same service, such as common large data component Hadoop, Hbase, Elasticsearch and other services. However, the multi-machine deployment mode has the problem of complex operation and maintenance, a common means for service operation and maintenance is to check logs, the check logs generally need to be logged in corresponding machine nodes for checking, and the efficiency is poor; at present, the logs are collected and displayed mainly in the following ways:
1. the log file is copied to the log center at regular time by using a system command combination of Secure Copy (SCP for short) + script and the like, and the mode is more original and has a plurality of defects: the efficiency is low, real-time log uploading cannot be realized, and a log transmission fault-tolerant mechanism is not provided.
2. The combination of various open source software such as ELK (elastic serach + Logstash + Kibana), flash + Kafka + elastic search + Kibana is used to realize the open source big data log collection mode, the advantage is that the collected logs can be displayed in real time, and the fault-tolerant mechanism is introduced into the logs, so that the defects that the whole process depends on more open source components and the deployment and maintenance costs are high are overcome.
3. The method has the disadvantages that the service of all service nodes shares one message queue, the log collection of all service nodes is influenced by the abnormal or blocked message queue network, and the fault-tolerant mechanism of log introduction is poor.
Therefore, a network client can display logs of service deployment services of different machine service nodes in real time, log contents can be refreshed manually or automatically, for a distributed system, the logs are collected to a center, and a log display server at the front end is necessary to check each service real-time log. Based on this, the embodiment of the present invention provides a method and an apparatus for presenting and aggregating logs of service nodes in a distributed system, and the technique can be applied to various distributed systems. To facilitate understanding of the embodiment, a detailed description will be given to a data processing method disclosed in the embodiment of the present invention.
The embodiment of the invention provides a method for displaying logs of service nodes in a distributed system, wherein the distributed system comprises a plurality of service nodes, at least one service program runs in each service node, the method is applied to a log display server, and the log display server is respectively in communication connection with the plurality of service nodes in the distributed system.
Referring to fig. 1, a distributed system includes a plurality of service nodes 102, where each service node 102 runs at least one service program, and a log display server 101 is respectively connected to the plurality of service nodes 102 in the distributed system in a communication manner.
The log display server is in communication connection with a plurality of service nodes in the distributed system respectively, and can be in communication connection through a Remote Procedure Call (RPC) protocol, so that the RPC framework actually provides a set of mechanism, so that the application programs can communicate with each other and conform to a server/client model; when the system is used, the client calls an interface provided by the server just like calling a local function; specifically, a GRPC protocol may be adopted, the GRPC has an advantage of a plurality of development languages supported by a cross-platform, and certainly, other RPC protocols may also be used to implement communication connection between the log presentation server and each service node in the distributed system. RPC is a protocol that requests services from a remote computer program over a network without knowledge of the underlying network technology. The RPC Protocol assumes the existence of some Transmission protocols, such as Transmission Control Protocol (TCP) or User Datagram Protocol (UDP), for carrying information data between communication programs. In an Open System Interconnection (OSI) network communication model, RPC spans a transport layer and an application layer, so that it is easier to develop an application program including a network distributed multi-program.
In other possible embodiments, the log display server and each service node in the distributed system may also implement communication connection by using a hypertext transfer Protocol (HTTP) or TCP.
The execution main body of the method is a log display server; as shown in fig. 2, the method comprises the steps of:
step S202, receiving log files sent by each service node, wherein the log files are running logs of corresponding service programs in the corresponding service nodes; the log file comprises log information and log content, wherein the log information comprises a service node name, a service program name and a log file name;
the service node can be understood as a log collection client; the log display server can receive a plurality of log files sent by a plurality of service nodes, wherein each log file is an operation log of a certain service program in each service node; the log content includes various types, such as logging in a system, upgrading software, modifying basic setting information, and the like; the service node name, the service program name and the log file name are generally automatically set and generated according to a system; service NODE name refers to the name of the current service NODE machine, e.g., NODE _ 1; the name of the SERVICE program refers to the name of which SERVICE program the current log file is about, such as SERVICE _ 1; log file names are used because a service program may have multiple log files, and each log file of the same service program has its own name, e.g., stdout. The log content refers to the text content of the log file.
Specifically, the log presentation server receives the log file in a format of < key1, key2, key3, value > through an RPC communication protocol, wherein the key1 is a service node name; key2 is the service name; key3 is the name of the log file; value is the log content.
Step S204, the log file is associated with the corresponding service program according to the log information;
since the log display server may receive a plurality of log files, the received log files need to be associated with the service program generating the log files according to the log information in the log files, so that when a user wants to obtain the log files generated by the service program, the corresponding log files can be displayed.
Specifically, the log file may be associated to a corresponding list of the log display server according to a service node name, a service program name, and the log file included in the log information.
A preferred embodiment:
analyzing the log information to obtain a service node name, a service program name and a log file name; according to the service node name, the log file is associated to a service node level list in a preset directory; according to the service program name, the log file is associated to a service program level list in a preset directory; and according to the name of the log file, associating the log file to a log file hierarchy list in a preset directory.
The pre-directory comprises a service node level, a service program level and a log file level, and the service node level, the service program level and the log file level respectively correspond to a service node name, a service program name and a log file name in log information; the log information can be analyzed by obtaining the key information of the first three of the received log files < key1, key2, key3, value >, and sequentially obtaining the service node name, the service program name, and the log file name corresponding to key1, key2, and key 3.
Specifically, in order to enable the log file to correspond to the log file requested to be viewed by the user, the log file may be associated to a service node hierarchy list in a preset directory according to the parsed service node name; according to the analyzed service program name, the log file is associated to a service program level list in a preset directory; and according to the name of the log file, associating the log file to a log file hierarchy list in a preset directory.
Step S206, when a log content viewing request of a user for a target service program in a target service node is obtained, a service program name list running in the target service node, a log file name list of the target service program and target log content are displayed according to the incidence relation between the log file and the service program.
The viewing request comprises a corresponding target service node and a target service program in the target service node; specifically, a target log file associated with a target service program in the target service node may be searched according to an association relationship between the log file and the service program, and the target log file obtained by the search may be displayed in a hierarchical directory manner.
For example, fig. 3 shows a directory of a log presentation server, where the log presentation server provides a front-end presentation service having an integrated service NODE machine list, in which service NODE names NODE _0, NODE _1, NODE _2, NODE _3, and NODE _4 of each service NODE machine are presented; an operation and maintenance person can check all SERVICE program names in a certain SERVICE NODE by clicking any one SERVICE NODE name, for example, after the operation and maintenance person clicks the SERVICE NODE name NODE _1, a log display SERVICE end displays all SERVICE program names SERVICE _0, SERVICE _1, SERVICE _2 and SERVICE _3 of the NODE _ 1; the operation and maintenance personnel can further click any one SERVICE program name to view all log file names of the SERVICE program, for example, after the operation and maintenance personnel click the SERVICE name SERVICE _1, the log display SERVICE end displays all log file names stdout. The operation and maintenance personnel can click any log file name, and then the log content of the log file can be obtained.
The method for displaying the logs of the service nodes in the distributed system is applied to a log display server and used for receiving log files sent by all the service nodes; associating the log file with a corresponding service program according to the log information included in the log file; and when the log content viewing request is acquired, displaying a service program name list operated in the target service node, a log file name list of the target service program and target log content according to the incidence relation between the log file and the service program. In the method, the distributed system comprises a plurality of service nodes, at least one service program runs in each service node, the log display service end is in communication connection with the service nodes in the distributed system respectively, the log file can be displayed to the log display service end in real time through the deployment method without complex components and with light weight, the deployment and maintenance cost is low, and meanwhile, the service operation and maintenance efficiency of the distributed system is improved.
The embodiment of the invention provides a log method for collecting service nodes in a distributed system, wherein the distributed system comprises a plurality of service nodes, at least one service program runs in each service node, each service node is in communication connection with a log display server, and the method is applied to the service nodes. As shown in fig. 4, the execution subject of the method is a service node, and the method includes the following steps:
step S402, acquiring a log file of a specified target service program in a configuration file according to the preset configuration file at intervals of a first preset period; the log file is an operation log of a corresponding service program in a corresponding service node; the log file comprises log information and log content, wherein the log information comprises a service node name, a service program name and a log file name;
the first preset period can be set between 1 and 30 seconds, and the specific duration can be determined according to actual needs; if the first preset period is short, the real-time performance of log updating is high, but the load on the whole distributed system is large; otherwise, the real-time performance of log updating is reduced, but the load of the whole distributed system is reduced. The preset configuration file is a computer file, can configure parameters and initial settings for some computer programs, and can perform different configuration files on different objects; specifically, according to actual requirements, if a log file of a certain service program is to be collected, the name of the service program may be configured into a configuration file.
The log file may include a plurality of log types, each determined by a different service, such as an application log, a Web and application server log, a garbage collector log, or a system log.
In this embodiment, taking setting to 1 second as an example for explanation, one possible way to obtain the log file may be to scan and obtain the log file of the target service program specified in the configuration file in a tail file (Tailfile) manner, and read the log file updated in the last first preset period line by line.
Step S404: and transmitting the log file to a log display server.
In this embodiment, the log file includes log information and log content, where the log information includes a service node name, a service program name, and a log file name.
Service NODE name refers to the name of the current service NODE machine, e.g., NODE _ 1; the name of the SERVICE program refers to the name of which SERVICE program the current log file is about, such as SERVICE _ 1; log file names are used because a service program may have multiple log files, and each log file of the same service has its own name, e.g., stdout. The log content refers to the text content of the log file.
When the service node transmits the log file to the log display service terminal, the log file can be transmitted through GRPC in the format of < key1, key2, key3 and value >; the keys 1, 2, 3 are generated based on the organization interface of the log display service end, and the keys 1, 2, 3 are respectively: the name of the service node, the name of the service program and the name of the log file, and the value is the log content. The service node can also be understood as a log collection client, and reports the log file to the log display server after collecting the log file in a first preset period of 1 second; and then displaying through the log display server.
In the method for aggregating service nodes and logs in a distributed system provided by the embodiment of the invention, each service node acquires the log file of the target service program appointed in the configuration file according to the preset configuration file at intervals of a first preset period, and then transmits the log file to the log display server, so that the log display server receives the log file capable of displaying each service node for the operation and maintenance personnel to screen and check. In the method, each service node is utilized to collect the log files in real time in a certain period and transmit the log files to the log display server, so that the log display server can display the log files to operation and maintenance personnel according to the preset directory, the efficiency of the operation and maintenance personnel for checking the log files is improved, the deployment and maintenance cost is low, and the technical problem of low service operation and maintenance efficiency of the existing distributed system is solved.
In a possible implementation manner, for a specific implementation manner of the step of acquiring the log file of the target service program specified in the configuration file in the step S402, the method includes:
step A1, determining a first target log file according to the target service program specified by the configuration file;
step A2, acquiring the number of newly added log records in the first target log file;
and step A3, acquiring the log records with the same number as the number of the pieces from the tail part of the first target log file according to the number of the pieces, and sending the log records as log files.
The types of the service programs are various, wherein some types of service programs can generate log records in real time and write the generated log records into log files; the log record in the first target log file is a real-time updated log file, and the first target log file is updated in real time according to the log record generated in real time by the service program, and the updating mode may be that new log records are sequentially added to the first target log file in a row mode. Therefore, in the case that the target service program generates a log record for a fact, in the embodiment, first, a first target log file may be determined according to the target service program specified by the configuration file; specifically, the number of newly added log records in the first target log file can be continuously scanned in a tail file (Tailfile) manner; according to the number, obtaining the log records with the same number as the number from the tail part of the first target log file as the log files and sending the log records to a log display end; in the method, the incremental log records are acquired in the incremental mode adaptively and are sent as the log file, and the newly generated log records can be displayed in real time aiming at the service program of the log records generated in real time.
In another possible implementation manner, for a specific implementation manner of the step of acquiring the log file of the target service program specified in the configuration file in step S402, the method further includes:
and determining a second target log file to be sent as a log file according to the target service program specified by the configuration file, wherein the second target log file is in a compressed packet format.
Because the types of the service programs are various, some service programs do not generate log records in real time, and generate a new log file compression packet at regular intervals; the second target log file is a log file generated by the service program of the medium type; in this case, therefore, the second target log file may be determined according to the target service program specified by the configuration file in the present embodiment; sending the determined second target log file serving as a log file to a log display end; in the method, the full log file is acquired in a full-scale mode and is sent as the log file, and the full-scale log record generated within a period of time can be displayed for the service program of the log record which cannot be generated in real time.
In another possible implementation manner, for a specific implementation manner of the step of acquiring the log file of the target service program specified in the configuration file in step S402, the method further includes:
determining the port number of the target service program according to the configuration file; monitoring the port number, acquiring a resource use log of the target service program to the service node, and sending the resource use log as a log file.
Because there are various types of service programs, some information of the service programs may not generate log files, for example, resource usage information (resources include resource usage of CPU, memory, network, and the like) of the service program to the service node where the service program is located, in this case, the present embodiment may determine the port number of the target service program according to the configuration file; the method comprises the steps that a port number is monitored, a resource use log of a target service program to a service node where the target service program is located can be obtained, the obtained resource use log serves as a log file and is sent to a log standing service terminal, in the method, resource use information of the service node is obtained by the port number of the service program adaptively and serves as the log file and is sent, the port number of the service program can be monitored in real time for the service program which cannot generate the log file, and the resource use information of the service node is displayed.
In a possible implementation manner, after the step S404, the following steps may be further included:
step S406: if the transmission of the log file fails, the log file is saved to a local temporary directory, and after every second preset period, the operation returns to step S404.
Through the steps, a fault-tolerant mechanism of log file transmission is realized, and when the GRPC communication network is abnormal, the log file is stored to a local temporary directory, namely the log file is landed and is retransmitted after the GRPC communication network is normal.
The second preset period can be set between 1 and 60 seconds, and the specific duration can also be determined according to actual needs. It should be noted that the second preset period may be equal to the first preset period, so that when the log file is collected in each first preset period, the second preset period is just the second preset period for retransmitting the log file.
Alternatively, the second preset period may be greater than the first preset period, for example, the second preset period is an integer multiple of the first preset period. When the second preset period is passed and the log files are to be retransmitted, a plurality of first preset periods are passed, and all the log files collected in the period are temporarily stored to the local. Or, after a plurality of second preset periods, the GRPC communication network returns to normal, and all log files collected during the period are temporarily stored to the local.
In a possible implementation manner, after the step S404, the following steps may be further included:
step S408: and if the log file is successfully transmitted, deleting the log file in the temporary directory.
If the log file is successfully transmitted to the log display server after the GRPC communication network is recovered to normal again, the log file previously stored in the temporary directory can be deleted, and the memory space is saved for the whole distributed system.
In the embodiment, at intervals of a first preset period, according to a preset configuration file, a log file of a specified target service program in the configuration file is acquired; transmitting the log file to a log display server; if the transmission of the log file fails, the log file is saved to a local temporary directory, and after a second preset period, the step of transmitting the log file to the log display server is returned; if the transmission of the log file is successful, deleting the log file in the temporary directory; in the method, each service node is used for collecting the log files in real time in a certain period and transmitting the log files to the log display server, so that the log display server can display the log files to operation and maintenance personnel in real time according to the preset directory, the efficiency of the operation and maintenance personnel for checking the log files is improved, and the technical problem of low service operation and maintenance efficiency of the existing distributed system is solved. And the dependence on open source components is less, and the deployment and maintenance costs are lower.
In addition, by setting a local temporary directory in the service node, the current log file can be stored in the temporary directory for disk-dropping under the condition of network failure, and retries can be performed after waiting for a certain time.
Corresponding to the foregoing method embodiment, an embodiment of the present invention further provides a device for displaying a log of a service node in a distributed system, where the distributed system includes a plurality of service nodes, each service node runs at least one service program, the device is disposed at a log display server, and the log display server is respectively in communication connection with the plurality of service nodes in the distributed system, as shown in fig. 5, the device includes:
a receiving module 51, configured to receive a log file sent by each service node, where the log file is an operation log of a corresponding service program in the corresponding service node; the log file comprises log information and log content, wherein the log information comprises a service node name, a service program name and a log file name;
the association module 52 is configured to associate the log file with the corresponding service program according to the log information;
the display module 53 is configured to, when a log content viewing request of a user for a target service program in a target service node is obtained, display a service program name list running in the target service node, a log file name list of the target service program, and target log content according to an association relationship between a log file and the service program.
The device for displaying the logs of the service nodes in the distributed system is arranged at a log display server and used for receiving log files sent by the service nodes; associating the log file with a corresponding service program according to the log information included in the log file; and when the log content viewing request is acquired, displaying a service program name list operated in the target service node, a log file name list of the target service program and target log content according to the incidence relation between the log file and the service program. In the method, the distributed system comprises a plurality of service nodes, at least one service program runs in each service node, the log display service end is in communication connection with the service nodes in the distributed system respectively, the log file can be displayed to the log display service end in real time through the deployment method without complex components and with light weight, the deployment and maintenance cost is low, and meanwhile, the service operation and maintenance efficiency of the distributed system is improved.
Further, the association module is further configured to: analyzing the log information to obtain a service node name, a service program name and a log file name; according to the service node name, the log file is associated to a service node level list in a preset directory; according to the service program name, the log file is associated to a service program level list in a preset directory; and according to the name of the log file, associating the log file to a log file hierarchy list in a preset directory.
The device for displaying the log of the service node in the distributed system provided by the embodiment of the invention has the same technical characteristics as the method for displaying the log of the service node in the distributed system provided by the embodiment, so that the same technical problems can be solved, and the same technical effect can be achieved.
Corresponding to the foregoing method embodiment, an embodiment of the present invention provides a device for collecting logs of service nodes in a distributed system, where the distributed system includes a plurality of service nodes, each service node runs at least one service program, each service node is in communication connection with a log display service end, and the device is disposed at a service node, and as shown in fig. 6, the device includes:
an obtaining module 61, configured to obtain, every interval of a first preset period, a log file of a target service program specified in a configuration file according to the preset configuration file; the log file is an operation log of a corresponding service program in a corresponding service node; the log file comprises log information and log content, wherein the log information comprises a service node name, a service program name and a log file name;
and the transmission module 62 is configured to transmit the log file to the log display server.
In the apparatus for aggregating service nodes and logs in a distributed system provided by the embodiment of the present invention, each service node acquires, at intervals of a first preset period, a log file of a target service program specified in a configuration file according to the preset configuration file, and then transmits the log file to a log display service end, so that the log display service end receives the log file capable of displaying each service node, and the log file is screened and checked by operation and maintenance personnel. In the method, each service node is used for collecting the log files in real time in a certain period and transmitting the log files to the log display server, so that the log display server can display the log files to operation and maintenance personnel according to the preset directory, the efficiency of the operation and maintenance personnel for checking the log files is improved, and the technical problem of low service operation and maintenance efficiency of the existing distributed system is solved.
Further, the obtaining module is further configured to: determining a first target log file according to a target service program specified by the configuration file; acquiring the number of newly added log records in a first target log file; and acquiring the log records with the same number as the number of the pieces from the tail part of the first target log file according to the number of the pieces, and sending the log records as log files.
Further, the obtaining module is further configured to: and determining a second target log file to be sent as a log file according to the target service program specified by the configuration file, wherein the second target log file is in a compressed packet format.
Further, the obtaining module is further configured to: determining the port number of the target service program according to the configuration file; monitoring the port number, acquiring a resource use log of the target service program to the service node, and sending the resource use log as a log file.
Further, the above apparatus is further configured to: and if the transmission of the log file fails, saving the log file to a local temporary directory, and returning to the step of transmitting the log file to the log display server after a second preset period.
Further, the above apparatus is further configured to: and if the log file is successfully transmitted, deleting the log file in the temporary directory.
The device for collecting logs of service nodes in the distributed system provided by the embodiment of the invention has the same technical characteristics as the method for collecting logs of service nodes in the distributed system provided by the embodiment, so that the same technical problems can be solved, and the same technical effect can be achieved.
As shown in fig. 7, an electronic device 700 according to an embodiment of the present invention includes a memory 701 and a processor 702, where the memory stores a computer program that is executable on the processor, and the processor executes the computer program to implement the steps of the method according to the above-mentioned embodiment.
As shown in fig. 7, the electronic device further includes: a bus 703 and a communication interface 704, and the processor 702, the communication interface 704, and the memory 701 are connected by the bus 703; the processor 702 is configured to execute executable modules, such as computer programs, stored in the memory 701.
The Memory 701 may include a high-speed Random Access Memory (RAM), and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 704 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.
Bus 703 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 7, but this does not indicate only one bus or one type of bus.
The memory 701 is used for storing a program, the processor 702 executes the program after receiving an execution instruction, and the method performed by the apparatus defined by the process disclosed in any of the foregoing embodiments of the present invention may be applied to the processor 702, or implemented by the processor 702.
The processor 702 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 702. The Processor 702 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component. The various methods, steps and logic blocks disclosed in the embodiments of the present invention may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 701, and the processor 702 reads the information in the memory 701, and completes the steps of the method in combination with the hardware thereof.
Corresponding to the method for presenting and collecting the logs of the service nodes in the distributed system, an embodiment of the present invention further provides a computer-readable storage medium, where machine executable instructions are stored in the computer-readable storage medium, and when the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to execute the steps of the method for presenting and collecting the logs of the service nodes in the distributed system.
The device for displaying and collecting the logs of the service nodes in the distributed system provided by the embodiment of the invention can be specific hardware on equipment or software or firmware installed on the equipment and the like. The device provided by the embodiment of the present invention has the same implementation principle and technical effect as the method embodiments, and for the sake of brief description, reference may be made to the corresponding contents in the method embodiments without reference to the device embodiments. It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the foregoing systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method can be implemented in other ways. The apparatus embodiments described above are merely illustrative, and for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
For another example, the division of the unit is only one division of logical functions, and there may be other divisions in actual implementation, and for another example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments provided by the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to perform all or part of the steps of the method for presenting and aggregating logs of service nodes in a distributed system according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus once an item is defined in one figure, it need not be further defined and explained in subsequent figures, and moreover, the terms "first", "second", "third", etc. are used merely to distinguish one description from another and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; and the modifications, changes or substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention. Are intended to be covered by the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (12)

1. A method for displaying logs of service nodes in a distributed system, wherein the distributed system comprises a plurality of service nodes, each service node runs at least one service program, the method is applied to a log display server, and the log display server is respectively in communication connection with the plurality of service nodes in the distributed system, and the method comprises the following steps:
receiving a log file sent by each service node, wherein the log file is an operation log of a corresponding service program in the corresponding service node; the log file comprises log information and log content, wherein the log information comprises a service node name, a service program name and a log file name;
associating the log file with a corresponding service program according to the log information;
when a log content viewing request of a user for a target service program in a target service node is acquired, a service program name list, a log file name list and target log content of the target service program which are operated in the target service node are displayed according to the incidence relation between the log file and the service program.
2. The method of claim 1, wherein associating the log file with a corresponding service according to the log information comprises:
analyzing the log information to obtain a service node name, a service program name and a log file name;
according to the service node name, the log file is associated to a service node level list in a preset directory;
according to the service program name, the log file is associated to a service program level list in a preset directory;
and according to the name of the log file, associating the log file to a log file hierarchy list in a preset directory.
3. A method for aggregating logs of service nodes in a distributed system, wherein the distributed system comprises a plurality of service nodes, each service node runs at least one service program therein, each service node is connected with a log display server in communication, and the method is applied to the service nodes, and the method comprises:
every interval of a first preset period, acquiring a log file of a specified target service program in a configuration file according to the preset configuration file; the log file is an operation log of a corresponding service program in a corresponding service node; the log file comprises log information and log content, wherein the log information comprises a service node name, a service program name and a log file name;
and transmitting the log file to the log display server.
4. The method of claim 3, wherein the step of obtaining the log file of the target service specified in the configuration file comprises:
determining a first target log file according to a target service program specified by the configuration file;
acquiring the number of newly added log records in the first target log file;
and acquiring the log records with the same number as the number from the tail part of the first target log file according to the number, and sending the log records as the log files.
5. The method of claim 3, wherein the step of obtaining the log file of the target service specified in the configuration file comprises:
and determining a second target log file to be sent as the log file according to the target service program specified by the configuration file, wherein the second target log file is in a compressed packet format.
6. The method of claim 3, wherein the step of obtaining the log file of the target service specified in the configuration file comprises:
determining the port number of the target service program according to the configuration file;
and monitoring the port number, acquiring a resource use log of the target service program to the service node, and sending the resource use log as the log file.
7. The method of claim 3, wherein after the step of transmitting the log file to the log presentation server, further comprising:
and if the transmission of the log file fails, saving the log file to a local temporary directory, and returning to the step of transmitting the log file to the log display server after a second preset period.
8. The method of claim 7, wherein after the step of transmitting the log file to the log presentation server, further comprising:
and if the transmission of the log file is successful, deleting the log file in the temporary directory.
9. An apparatus for displaying a log of a service node in a distributed system, wherein the distributed system includes a plurality of service nodes, at least one service program runs in each service node, the apparatus is disposed at a log display server, and the log display server is respectively communicatively connected to the plurality of service nodes in the distributed system, the apparatus includes:
the receiving module is used for receiving log files sent by each service node, wherein the log files are running logs of corresponding service programs in the corresponding service nodes; the log file comprises log information and log content, wherein the log information comprises a service node name, a service program name and a log file name;
the association module is used for associating the log file with a corresponding service program according to the log information;
and the display module is used for displaying a service program name list, a log file name list of the target service program and target log content which are operated in the target service node according to the incidence relation between the log file and the service program when a log content viewing request of a user for the target service program in the target service node is obtained.
10. An apparatus for aggregating logs of service nodes in a distributed system, wherein the distributed system comprises a plurality of service nodes, each service node runs at least one service program therein, each service node is communicatively connected to a log display server, the apparatus is disposed at the service node, and the apparatus comprises:
the acquisition module is used for acquiring a log file of a specified target service program in a configuration file according to the preset configuration file every a first preset period; the log file is an operation log of a corresponding service program in a corresponding service node; the log file comprises log information and log content, wherein the log information comprises a service node name, a service program name and a log file name;
and the transmission module is used for transmitting the log file to the log display server.
11. An electronic device comprising a memory and a processor, wherein the memory stores a computer program operable on the processor, and wherein the processor implements the method for presenting a log of service nodes in a distributed system according to any of the preceding claims 1 to 2, or the method for aggregating logs of service nodes in a distributed system according to any of the preceding claims 3 to 8 when executing the computer program.
12. A computer readable storage medium having stored thereon machine executable instructions which, when invoked and executed by a processor, cause the processor to perform the steps of the method of exposing a log of service nodes in a distributed system of any of claims 1 to 2 or the steps of the method of aggregating logs of service nodes in a distributed system of any of claims 3 to 8.
CN202010570955.6A 2020-06-19 2020-06-19 Method and device for showing and collecting logs of service nodes in distributed system Active CN111737207B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010570955.6A CN111737207B (en) 2020-06-19 2020-06-19 Method and device for showing and collecting logs of service nodes in distributed system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010570955.6A CN111737207B (en) 2020-06-19 2020-06-19 Method and device for showing and collecting logs of service nodes in distributed system

Publications (2)

Publication Number Publication Date
CN111737207A true CN111737207A (en) 2020-10-02
CN111737207B CN111737207B (en) 2024-04-09

Family

ID=72651951

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010570955.6A Active CN111737207B (en) 2020-06-19 2020-06-19 Method and device for showing and collecting logs of service nodes in distributed system

Country Status (1)

Country Link
CN (1) CN111737207B (en)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112417009A (en) * 2020-10-12 2021-02-26 南京江北新区生物医药公共服务平台有限公司 Visual display method and device for medical data
CN113190522A (en) * 2021-06-29 2021-07-30 阿里云计算有限公司 Log processing method and device, computing equipment and storage medium
CN114793193A (en) * 2022-04-22 2022-07-26 深圳市东晟数据有限公司 Rapid association classification and hierarchical storage method for network security logs
CN115442226A (en) * 2022-08-01 2022-12-06 远光软件股份有限公司 Log collection method, related device and storage medium
CN115442226B (en) * 2022-08-01 2024-06-11 远光软件股份有限公司 Log acquisition method, related device and storage medium

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040167912A1 (en) * 2003-02-20 2004-08-26 International Business Machines Corporation Unified logging service for distributed applications
US20110040811A1 (en) * 2009-08-17 2011-02-17 International Business Machines Corporation Distributed file system logging
CN105205167A (en) * 2015-10-10 2015-12-30 国网信息通信产业集团有限公司 Log data system
CN106227727A (en) * 2016-06-30 2016-12-14 乐视控股(北京)有限公司 Daily record update method, device and the system of a kind of distributed system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040167912A1 (en) * 2003-02-20 2004-08-26 International Business Machines Corporation Unified logging service for distributed applications
US20110040811A1 (en) * 2009-08-17 2011-02-17 International Business Machines Corporation Distributed file system logging
CN105205167A (en) * 2015-10-10 2015-12-30 国网信息通信产业集团有限公司 Log data system
CN106227727A (en) * 2016-06-30 2016-12-14 乐视控股(北京)有限公司 Daily record update method, device and the system of a kind of distributed system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
张煜;: "一种使用Node.js构建的分布式数据流日志服务系统", 计算机系统应用, no. 02 *
王宇飞;刘丹;吴嘉生;: "基于OSGi的分布式系统集中日志管理方案", 计算机系统应用, no. 06 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112417009A (en) * 2020-10-12 2021-02-26 南京江北新区生物医药公共服务平台有限公司 Visual display method and device for medical data
CN113190522A (en) * 2021-06-29 2021-07-30 阿里云计算有限公司 Log processing method and device, computing equipment and storage medium
CN114793193A (en) * 2022-04-22 2022-07-26 深圳市东晟数据有限公司 Rapid association classification and hierarchical storage method for network security logs
CN115442226A (en) * 2022-08-01 2022-12-06 远光软件股份有限公司 Log collection method, related device and storage medium
CN115442226B (en) * 2022-08-01 2024-06-11 远光软件股份有限公司 Log acquisition method, related device and storage medium

Also Published As

Publication number Publication date
CN111737207B (en) 2024-04-09

Similar Documents

Publication Publication Date Title
US9961129B2 (en) Business transaction correlation with client request monitoring data
CN111737207B (en) Method and device for showing and collecting logs of service nodes in distributed system
US8156216B1 (en) Distributed data collection and aggregation
CN111078504A (en) Distributed call chain tracking method and device, computer equipment and storage medium
CN107370806B (en) HTTP status code monitoring method, device, storage medium and electronic equipment
CN110309029B (en) Abnormal data acquisition method and device, computer equipment and storage medium
CN103078945B (en) To the method and system that browser crash data processes
CN104104707A (en) Webpage data update method, terminal, cache server and system
CN111694797A (en) File uploading and analyzing method, device, server and medium
CN106339273A (en) Application program restoration method, terminal and server
US10613922B2 (en) Synchronously generating diagnostic data
CN109559121B (en) Transaction path call exception analysis method, device, equipment and readable storage medium
US10432490B2 (en) Monitoring single content page application transitions
US20070094266A1 (en) Collecting information on component implementation and use
WO2015195407A1 (en) Method and apparatus for monitoring and determining page load times
CN111385157B (en) Server abnormity detection method and device
CN115473858A (en) Data transmission method and streaming data transmission system
CN110875832B (en) Abnormal service monitoring method, device and system and computer readable storage medium
US20170223136A1 (en) Any Web Page Reporting and Capture
CN111698109A (en) Method and device for monitoring log
CN112416698B (en) Expansion method and device of monitoring system, storage medium and electronic equipment
CN115543491A (en) Microservice processing method and device
CN113141366B (en) Message submitting method and device, storage medium and electronic equipment
CN111258964A (en) Log processing method and device, storage medium and electronic equipment
CN113778800B (en) Error information processing method, device, system, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant