CN111176951A - Log output configuration/log processing method/system, medium, and server - Google Patents
Log output configuration/log processing method/system, medium, and server Download PDFInfo
- Publication number
- CN111176951A CN111176951A CN201911414044.8A CN201911414044A CN111176951A CN 111176951 A CN111176951 A CN 111176951A CN 201911414044 A CN201911414044 A CN 201911414044A CN 111176951 A CN111176951 A CN 111176951A
- Authority
- CN
- China
- Prior art keywords
- server
- log
- service access
- access log
- output
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 238000000034 method Methods 0.000 claims abstract description 58
- 238000004891 communication Methods 0.000 claims abstract description 20
- 238000012545 processing Methods 0.000 claims description 35
- 238000004590 computer program Methods 0.000 claims description 11
- 238000001914 filtration Methods 0.000 claims description 10
- 238000012986 modification Methods 0.000 claims description 8
- 230000004048 modification Effects 0.000 claims description 8
- 238000004806 packaging method and process Methods 0.000 claims description 5
- 238000012544 monitoring process Methods 0.000 claims description 3
- 239000003795 chemical substances by application Substances 0.000 description 20
- 230000004044 response Effects 0.000 description 14
- 238000012546 transfer Methods 0.000 description 12
- 230000008569 process Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000013079 data visualisation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000002904 solvent Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
Abstract
The invention provides a log output configuration/log processing method/system, a medium and a server, wherein the log output configuration method is applied to a first server, wherein the first server is used for collecting a service access log of a microservice running on the first server; the first server is in communication connection with a second server; the configuration method of the log collection service comprises the following steps: modifying a configuration file preset in the first server to designate a second server to receive the service access log of the microservice; the configuration file is used for providing an external load balancing agent; enabling a default log output format; repackaging the modified configuration file and restarting; and pushing the service access log to a specified second server based on the modified configuration file. The method and the system can count and display the service access data of the micro-service in real time, and can quickly position the micro-service with performance bottleneck and hidden trouble.
Description
Technical Field
The present invention belongs to the field of communication technology, and is especially log output configuring and log processing method and system, medium and server.
Background
With the gradual maturity of cloud computing and a large number of open source lightweight technologies, a new architecture design style, namely a micro-service architecture, is promoted. The micro-service architecture is characterized by fine-grained service boundary and is beneficial to independent development, test, deployment and expansion. However, as the service size becomes larger and larger, the service call chain becomes longer and the performance problem becomes more and more important. The intuitive performance of a service is the response time and throughput of the request.
In the prior art, the micro-services on the platform are quickly positioned by counting and analyzing results, wherein the micro-services have performance bottlenecks and hidden problems. Therefore, how to provide a method/system, medium, and server for log output configuration/log processing to solve the problem that the prior art cannot locate which micro-services have performance bottlenecks and hidden problems is a technical problem to be solved urgently by those skilled in the art.
Disclosure of Invention
In view of the above drawbacks of the prior art, an object of the present invention is to provide a method/system, a medium, and a server for configuring/processing a log output, which are used to solve the problem that the prior art cannot locate which micro-services have performance bottlenecks and hidden problems.
To achieve the above and other related objects, an aspect of the present invention provides a communication network, comprising: the first server is used for collecting a service access log of the micro service operated on the first server and pushing the service access log in an appointed way; the second server is used for collecting the service access log pushed by the first server and performing filtering formatting processing on the service access log; the search server is used for forwarding the service access log pushed by the second server; and the display is used for receiving the service access log forwarded by the search server and counting and displaying the service access log in real time.
The invention provides a configuration method of log output, which is applied to a first server, wherein the first server is used for collecting a service access log of a microservice running on the first server; the first server is in communication connection with a second server; the configuration method of the log collection service comprises the following steps: modifying a configuration file preset in the first server to designate a second server to receive the service access log of the microservice; the configuration file is used for providing an external load balancing agent; enabling a default log output format; repackaging the modified configuration file and restarting; and pushing the service access log to a specified second server based on the modified configuration file.
In an embodiment of the present invention, the step of modifying the configuration file preset in the first server includes adding the address and the port of the receiving end to the configuration file.
The invention also provides a log processing method based on the configuration method of the log output, which is applied to a second server, wherein the second server is respectively communicated with the first server and is connected with a search server; the processing method of the log comprises the following steps: receiving a service access log pushed by the first server; filtering and formatting the service access log to form a service access log which can be written into the search server; and outputting the service access log which can be written into the search server to a designated search server.
In an embodiment of the present invention, the step of receiving the service access log pushed by the first server includes: defining a monitoring port for receiving the service access log; and defining a log label for the service access log.
In an embodiment of the present invention, the step of outputting the processed service access log to a designated search server includes outputting the service access log that can be written in the search server to an address and a port of the search server.
The invention further provides a configuration system of log output, which is applied to a first server, wherein the first server is used for collecting service access logs of the micro-service running on the first server; the first server is in communication connection with a second server; the log output configuration system comprises: the modification module is used for modifying the configuration file which is preset in the first server so as to designate a second server which is to receive the service access log of the microservice; the configuration file is used for providing an external load balancing agent; the starting module is used for starting a default log output format; the packaging module is used for repackaging the modified configuration files and restarting the configuration files; and the pushing module is used for pushing the service access log to a specified second server based on the modified configuration file.
The invention also provides a log processing system for a configuration system based on log output, which is applied to a second server, wherein the second server is respectively connected with the first server and a search server; the processing system of the log comprises: the receiving module is used for receiving the service access log pushed by the first server; the processing module is used for filtering and formatting the service access log to form a service access log which can be written into the search server; and the output module is used for outputting the processed service access log to a specified search server.
Still another aspect of the present invention provides a storage medium having stored thereon a computer program that, when executed by a processor, implements the method of configuring the log output or the method of processing a log that implements the method of configuring based on the log output.
A final aspect of the present invention provides a server comprising: a processor and a memory; the memory is used for storing a computer program, and the processor is used for executing the computer program stored by the memory so as to enable the server to execute the configuration method of log output or execute the processing method of the log based on the configuration method of the log output.
As described above, the configuration/log processing method/system, medium, and server for log output according to the present invention have the following advantages:
the method and the system can count and display the service access data of the micro-service in real time, and can quickly position the micro-service with performance bottleneck and hidden trouble.
Drawings
Fig. 1 is a schematic view of a communication network to which the present invention is applied.
Fig. 2 is a flowchart illustrating a configuration method of log output according to an embodiment of the present invention.
Fig. 3 is a flowchart illustrating a log processing method according to an embodiment of the configuration method based on log output.
FIG. 4 is a schematic diagram of the interaction between the configuration system for log output and the log processing system based on the configuration system for log output according to the present invention.
Description of the element reference numerals
Detailed Description
The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.
It should be noted that the drawings provided in the following embodiments are only for illustrating the basic idea of the present invention, and the components related to the present invention are only shown in the drawings rather than drawn according to the number, shape and size of the components in actual implementation, and the type, quantity and proportion of the components in actual implementation may be changed freely, and the layout of the components may be more complicated.
Example one
The embodiment provides a configuration method of log output, which is characterized in that the configuration method is applied to a first server, wherein the first server is used for collecting a service access log of a microservice running on the first server; the first server is in communication connection with a second server; the configuration method of the log collection service comprises the following steps:
modifying a configuration file preset in the first server to designate a second server to receive the service access log of the microservice; the configuration file is used for providing an external load balancing agent;
enabling a default log output format;
repackaging the modified configuration file and restarting;
and pushing the service access log to a specified second server based on the modified configuration file.
The embodiment also provides a log processing method based on the configuration method of the log output, which is applied to a second server, wherein the second server is respectively communicated with the first server and connected with a search server; the processing method of the log comprises the following steps:
receiving a service access log pushed by the first server;
filtering and formatting the service access log to form a service access log which can be written into the search server;
and outputting the service access log which can be written into the search server to a designated search server.
The following describes in detail a configuration method of log output and a processing method of a log based on the configuration method of log output provided by the present embodiment with reference to the drawings.
Please refer to fig. 1, which is a schematic view of a communication network in which the present invention is applied. As shown in fig. 1, the communication network 1 includes at least one first server 11, at least one second server 12, a search server 13 and a display 14.
The first server is used for collecting a service access log of the micro service operated on the first server and pushing the service access log in an appointed mode. In this embodiment, the first server is provided with an external load balancing agent Marathon-lb of the DC/OS micro-service cluster, so as to forward the external access request to the internal micro-service, and the bottom layer is HAProxy.
At least one second server 12 connected to the first server 11 in a communication manner is configured to collect the service access log pushed by the first server, and perform filtering and formatting processing on the service access log. In this embodiment, the second server 12 filters and formats the collected service access logs, processes the service access logs into content with higher readability, and writes the content into the search server 13. In this embodiment, the second server 12 is provided with a collection tool logstack for collecting the service access log.
The search server 13 connected to the second server 12 in communication is configured to forward the service access log pushed by the second server. In this embodiment, the search server 13 is a Lucene-based search server, and is used to provide a full-text search engine with distributed multi-user capability.
And the display 14 which is in communication connection with the search server 13 is used for receiving the service access log forwarded by the search server, counting and displaying the service access log in real time. In the present embodiment, the display 14 is provided with an open source data visualization and mining tool (for example, Kibana, on which data views and dashboards defining the data to be displayed are configured) so as to count and display the service access logs in real time through Kibana.
Please refer to fig. 2, which is a flowchart illustrating a configuration method for log output according to an embodiment. As shown in fig. 2, the configuration method of log output specifically includes the following steps:
s21, entering a configuration directory, and searching a configuration file preset in the first server. The configuration file is used to provide an external load balancing agent. The configuration file is, in this embodiment, a HAPROXY _ HEAD configuration file.
S22, modifying the configuration file to specify a second server to receive the service access log of the micro service.
In this embodiment, the step S22 includes adding the address of the receiving end and the port address to the configuration file.
In particular, the amount of the solvent to be used,
global
log 10.10.10.1:5003local3 info. Wherein, the 10.10.10.1:5003 part is the second server address and the port.
S23, the default http log output format is enabled.
S24, repackaging the modified configuration files into tgz compressed packages, and recording new configuration files by restarting Marathon-lb.
Please refer to fig. 3, which is a flowchart illustrating a log processing method based on a configuration method of log output according to an embodiment. As shown in fig. 3, the method for processing the log based on the configuration method of the log output specifically includes the following steps:
s31, receiving the service access log pushed by the first server.
Specifically, the S31 includes:
and defining a monitoring port for receiving the service access log.
For example, the listening port is defined as 5003.
And defining a log label for the service access log.
S32, filtering and formatting the service access log to form a service access log which can be written into the search server. In this embodiment, the output field of the service access log includes:
a Client _ ip, which is described as a front-end Client address;
frontend _ name, which is specified as the HAPROXY snoop name;
a backup _ name, which is described as a backend server name;
time _ request, which specifies the latency before the front-end client HTTP requests the load agent response
time _ backup _ connect, which is indicated as the connection backend server waiting response time;
time _ backup _ response, which is described as the backend server HTTP request result response time;
time _ duration, which specifies the complete response time for the client to request the HAPROXY result;
HTTP _ status _ code, which is specified as HTTP response status code;
bytes _ read, this field states the client requests the result packet size;
actconn, which is used for describing the number of concurrent connections of the process during the session;
fenconn, which field indicates the number of HAPROXY concurrent connections at the time of recording a session;
the field of becon indicates the number of concurrent connections of the back-end server during session recording;
retries, which is indicated as the number of times of retry of the HAPROXY connection back-end server;
srv _ queue, which specifies the number of queues HAPROXY is in responding to the HTTP request;
backend _ queue, which indicates the number of queues of the back-end server before responding to the HTTP request;
captured _ response _ headers, which specifies the HTTP address requested by the client
HTTP _ method, this field is specified as the client HTTP request method;
http _ request, this field is illustrated as a client request.
The S32 includes:
matching the log field of the service access log, and defining a log output field name;
intercepting the field content in the service access log before the request field of the front-end client;
deleting the field content which is not required to be output in the service access log;
intercepting the content in the service access log before the name field of the back-end server;
defining the output of a waiting time field before a hypertext transfer protocol request load agent of a front-end client end responds as an integer;
defining the output of a waiting response time field of a connection back-end server as an integer;
defining the response time field output of the hypertext transfer protocol request load agent result of the back-end server as an integer;
defining the output of the hypertext transfer protocol request load agent result response time field of the front-end client as an integer;
defining the size field output of a hypertext transfer protocol request result data packet of a front-end client as an integer;
defining the output of the concurrent connection digital field of the process as an integer when the session is recorded;
defining the output of the load agent concurrent connection digital field as an integer when recording the session;
defining the output of the concurrent connection digital section of the back-end server as an integer when recording the session;
defining the retry number field output of the load agent connected with the back-end server as an integer;
defining the queue number field output of the load agent before responding to the hypertext transfer protocol request as an integer; and/or
And defining the queue number field output of the back-end server before responding to the hypertext transfer protocol request as an integer.
And S33, outputting the service access log which can be written in the search server to the designated search server.
Specifically, the service access log which can be written into the search server is output to the address and the port of the search server. In this embodiment, the service access log that can be written into the search server is a log that passes through a matching log field, an intercepting log field, a deleting log field that does not need to be output, and a defining field that is output as an integer.
For example, the address and port of the search server is http://10.10.10.2: 9200.
The present embodiment also provides a storage medium (also referred to as a computer-readable storage medium) having stored thereon a computer program that, when executed by a processor, implements the above-described configuration method of log output or a processing method of a log that implements the above-described configuration method based on log output.
Those of ordinary skill in the art will understand that: all or part of the steps for implementing the above method embodiments may be performed by hardware associated with a computer program. The aforementioned computer program may be stored in a computer readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
The method for configuring the log output and the method for processing the log based on the method for configuring the log output can be used for rapidly positioning the micro-service with performance bottleneck and hidden trouble problems.
Example two
The embodiment provides a configuration system for log output, which is characterized in that the configuration system is applied to a first server, wherein the first server is used for collecting a service access log of a microservice running on the first server; the first server is in communication connection with a second server; the log output configuration system comprises:
the modification module is used for modifying the configuration file which is preset in the first server so as to designate a second server which is to receive the service access log of the microservice; the configuration file is used for providing an external load balancing agent;
the starting module is used for starting a default log output format;
the packaging module is used for repackaging the modified configuration files and restarting the configuration files;
and the pushing module is used for pushing the service access log to a specified second server based on the modified configuration file.
The embodiment also provides a log processing system for a configuration system based on log output, which is applied to a second server, wherein the second server is respectively communicated with the first server and connected with a search server; the processing system of the log comprises:
the receiving module is used for receiving the service access log pushed by the first server;
the processing module is used for filtering and formatting the service access log to form a service access log which can be written into the search server;
and the output module is used for outputting the processed service access log to a specified search server.
The configuration system of log output and the processing system of the log based on the configuration system of log output of the present embodiment will be described in detail with reference to the drawings. Referring to fig. 4, a schematic structural diagram of the interaction between the configuration system for log output and the log processing system based on the configuration system for log output is shown. As shown in fig. 4, the configuration system 41 for log output includes a directory entry module 411, a modification module 412, an enabling module 413, a packaging module 414, and a pushing module 415.
The directory entry module 411 is configured to enter a configuration directory, and search for a configuration file preset in the first server. The configuration file is used to provide an external load balancing agent. The configuration file is, in this embodiment, a HAPROXY _ HEAD configuration file.
The modification module 412 is configured to modify the configuration file to specify a second server to receive the service access log of the microservice.
In this embodiment, the step S22 includes adding the address of the receiving end and the port address to the configuration file.
The enabling module 413 is configured to enable a default http log output format.
The packaging module 414 is used to repackage the modified configuration file into tgz compressed packages and record a new configuration file for the restart Marathon-lb.
The pushing module 415 is configured to push the service access log to a specified second server based on the modified configuration file.
With continued reference to fig. 4, the processing system 42 for configuring the system log based on log output includes a receiving module 421, a processing module 422, and an outputting module 423.
The receiving module 421 is configured to receive the service access log pushed by the first server.
Specifically, the receiving module 421 is further configured to define a listening port for receiving the service access log, and define a log tag for the service access log.
The processing module 422 is configured to filter and format the service access log to form a service access log that can be written to the search server.
The processing module 422 is specifically configured to:
matching the log field of the service access log, and defining a log output field name;
intercepting the field content in the service access log before the request field of the front-end client;
deleting the field content which is not required to be output in the service access log;
intercepting the content in the service access log before the name field of the back-end server;
defining the output of a waiting time field before a hypertext transfer protocol request load agent of a front-end client end responds as an integer;
defining the output of a waiting response time field of a connection back-end server as an integer;
defining the response time field output of the hypertext transfer protocol request load agent result of the back-end server as an integer;
defining the output of the hypertext transfer protocol request load agent result response time field of the front-end client as an integer;
defining the size field output of a hypertext transfer protocol request result data packet of a front-end client as an integer;
defining the output of the concurrent connection digital field of the process as an integer when the session is recorded;
defining the output of the load agent concurrent connection digital field as an integer when recording the session;
defining the output of the concurrent connection digital section of the back-end server as an integer when recording the session;
defining the retry number field output of the load agent connected with the back-end server as an integer;
defining the queue number field output of the load agent before responding to the hypertext transfer protocol request as an integer; and/or
And defining the queue number field output of the back-end server before responding to the hypertext transfer protocol request as an integer.
The output module 423 is configured to output the service access log that can be written to the search server to a specified search server.
Specifically, the output module 423 outputs the service access log which can be written into the search server to the address and the port of the search server.
It should be noted that the division of the modules of the above system is only a logical division, and the actual implementation may be wholly or partially integrated into one physical entity, or may be physically separated. And the modules can be realized in a form that all software is called by the processing element, or in a form that all the modules are realized in a form that all the modules are called by the processing element, or in a form that part of the modules are called by the hardware. For example: the x module can be a separately established processing element, and can also be integrated in a certain chip of the system. In addition, the x-module may be stored in the memory of the system in the form of program codes, and may be called by one of the processing elements of the system to execute the functions of the x-module. Other modules are implemented similarly. All or part of the modules can be integrated together or can be independently realized. The processing element described herein may be an integrated circuit having signal processing capabilities. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in the form of software. These above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors (DSPs), one or more Field Programmable Gate Arrays (FPGAs), and the like. When a module is implemented in the form of a processing element scheduler code, the processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. These modules may be integrated together and implemented in the form of a System-on-a-chip (SOC).
EXAMPLE III
This embodiment provides a server, including first server and second server, first server and second server include: a processor, memory, transceiver, communication interface, or/and system bus; the memory is used for storing computer programs, the communication interface is used for communicating with other devices, and the processor and the transceiver are used for running the computer programs, so that the first server executes each step of the configuration method based on log output, and the second server executes the processing method based on the log of the configuration method based on the log output.
The above-mentioned system bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The system bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus. The communication interface is used for realizing communication between the database access device and other equipment (such as a client, a read-write library and a read-only library). The Memory may include a Random Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory.
The Processor may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the integrated Circuit may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, or discrete hardware components.
The protection scope of the method of the present invention is not limited to the execution sequence of the steps illustrated in the embodiment, and all the solutions implemented by adding, subtracting and replacing steps in the prior art according to the principle of the present invention are included in the protection scope of the present invention.
The present invention also provides a system, which can implement the method of the present invention, but the implementation device of the method of the present invention includes, but is not limited to, the structure of the system illustrated in the present embodiment, and all the structural modifications and substitutions of the prior art made according to the principle of the present invention are included in the protection scope of the present invention.
In conclusion, the method and the device can count and display the service access data of the micro-service in real time, and can quickly locate the micro-service with performance bottleneck and hidden trouble. The invention effectively overcomes various defects in the prior art and has high industrial utilization value.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.
Claims (10)
1. A communication network, comprising:
the first server is used for collecting a service access log of the micro service operated on the first server and pushing the service access log in an appointed way;
the second server is used for collecting the service access log pushed by the first server and performing filtering formatting processing on the service access log;
the search server is used for forwarding the service access log pushed by the second server;
and the display is used for receiving the service access log forwarded by the search server and counting and displaying the service access log in real time.
2. The log output configuration method is applied to a first server, wherein the first server is used for collecting service access logs of micro services running on the first server; the first server is in communication connection with a second server; the configuration method of the log collection service comprises the following steps:
modifying a configuration file preset in the first server to designate a second server to receive the service access log of the microservice; the configuration file is used for providing an external load balancing agent;
enabling a default log output format;
repackaging the modified configuration file and restarting;
and pushing the service access log to a specified second server based on the modified configuration file.
3. The method according to claim 2, wherein the step of modifying the configuration file pre-installed in the first server comprises adding an address and a port of the receiving end to the configuration file.
4. A log processing method based on a configuration method of log output of any one of a right 2 and a right 3 is characterized by being applied to a second server, wherein the second server is respectively communicated with the first server and connected with a search server; the processing method of the log comprises the following steps:
receiving a service access log pushed by the first server;
filtering and formatting the service access log to form a service access log which can be written into the search server;
and outputting the service access log which can be written into the search server to a designated search server.
5. The log processing method according to claim 4, wherein the step of receiving the service access log pushed by the first server includes:
defining a monitoring port for receiving the service access log;
and defining a log label for the service access log.
6. The method according to claim 4, wherein the step of outputting the processed service access log to a designated search server comprises outputting the service access log writable to the search server to an address and a port of the search server.
7. The log output configuration system is applied to a first server, wherein the first server is used for collecting service access logs of micro services running on the first server; the first server is in communication connection with a second server; the log output configuration system comprises:
the modification module is used for modifying the configuration file which is preset in the first server so as to designate a second server which is to receive the service access log of the microservice; the configuration file is used for providing an external load balancing agent;
the starting module is used for starting a default log output format;
the packaging module is used for repackaging the modified configuration files and restarting the configuration files;
and the pushing module is used for pushing the service access log to a specified second server based on the modified configuration file.
8. A processing system for configuring the log of the system based on the log output of the right 7 is characterized by being applied to a second server, wherein the second server is respectively communicated with the first server and connected with a search server; the processing system of the log comprises:
the receiving module is used for receiving the service access log pushed by the first server;
the processing module is used for filtering and formatting the service access log to form a service access log which can be written into the search server;
and the output module is used for outputting the processed service access log to a specified search server.
9. A storage medium on which a computer program is stored, the program, when executed by a processor, implementing a method of configuring a log output according to any one of claims 2 to 3 or a method of processing a log implementing a method of configuring a log based on a log output according to any one of claims 4 to 6.
10. A server, comprising: a processor and a memory;
the memory is used for storing a computer program, and the processor is used for executing the computer program stored by the memory to enable the server to execute the configuration method of log output according to any one of claims 1 to 2 or execute the processing method of the log based on the configuration method of log output according to any one of claims 3 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911414044.8A CN111176951A (en) | 2019-12-31 | 2019-12-31 | Log output configuration/log processing method/system, medium, and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911414044.8A CN111176951A (en) | 2019-12-31 | 2019-12-31 | Log output configuration/log processing method/system, medium, and server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111176951A true CN111176951A (en) | 2020-05-19 |
Family
ID=70654398
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911414044.8A Pending CN111176951A (en) | 2019-12-31 | 2019-12-31 | Log output configuration/log processing method/system, medium, and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111176951A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106534257A (en) * | 2016-09-29 | 2017-03-22 | 国家电网公司 | Multi-level cluster-type construction multi-source safety log collection system and method |
| CN107463602A (en) * | 2017-06-15 | 2017-12-12 | 努比亚技术有限公司 | A kind of log processing method and server, client |
| US20170359424A1 (en) * | 2016-06-09 | 2017-12-14 | International Business Machines Corporation | Dynamic generation of network routing configuration with service requirements |
| CN108847989A (en) * | 2018-06-29 | 2018-11-20 | 杭州安恒信息技术股份有限公司 | Log processing method, business service system and electronic equipment based on micro services framework |
| CN108989101A (en) * | 2018-07-04 | 2018-12-11 | 北京奇艺世纪科技有限公司 | A kind of log output system, method and electronic equipment |
| CN109960621A (en) * | 2017-12-22 | 2019-07-02 | 南京欣网互联网络科技有限公司 | A kind of data pick-up method based on big data visual control platform |
| CN110175154A (en) * | 2019-05-17 | 2019-08-27 | 深圳前海微众银行股份有限公司 | A kind of processing method of log recording, server and storage medium |
| CN110245060A (en) * | 2019-05-20 | 2019-09-17 | 北京奇艺世纪科技有限公司 | Service monitoring method and equipment, service monitoring device and medium |
-
2019
- 2019-12-31 CN CN201911414044.8A patent/CN111176951A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170359424A1 (en) * | 2016-06-09 | 2017-12-14 | International Business Machines Corporation | Dynamic generation of network routing configuration with service requirements |
| CN106534257A (en) * | 2016-09-29 | 2017-03-22 | 国家电网公司 | Multi-level cluster-type construction multi-source safety log collection system and method |
| CN107463602A (en) * | 2017-06-15 | 2017-12-12 | 努比亚技术有限公司 | A kind of log processing method and server, client |
| CN109960621A (en) * | 2017-12-22 | 2019-07-02 | 南京欣网互联网络科技有限公司 | A kind of data pick-up method based on big data visual control platform |
| CN108847989A (en) * | 2018-06-29 | 2018-11-20 | 杭州安恒信息技术股份有限公司 | Log processing method, business service system and electronic equipment based on micro services framework |
| CN108989101A (en) * | 2018-07-04 | 2018-12-11 | 北京奇艺世纪科技有限公司 | A kind of log output system, method and electronic equipment |
| CN110175154A (en) * | 2019-05-17 | 2019-08-27 | 深圳前海微众银行股份有限公司 | A kind of processing method of log recording, server and storage medium |
| CN110245060A (en) * | 2019-05-20 | 2019-09-17 | 北京奇艺世纪科技有限公司 | Service monitoring method and equipment, service monitoring device and medium |
Non-Patent Citations (2)
| Title |
|---|
| 张振 等: "基于微服务架构的日志监控系统的设计与实现", 《软件》, vol. 38, no. 11, pages 196 - 201 * |
| 王宇飞;刘丹;吴嘉生;: "基于OSGi的分布式系统集中日志管理方案", 计算机系统应用, no. 06 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2020203145B2 (en) | Processing data from multiple sources | |
| WO2020238066A1 (en) | Software pressure test method and apparatus, and storage medium and server | |
| US9391831B2 (en) | Dynamic stream processing within an operator graph | |
| US9372780B2 (en) | Breakpoint continuation for stream computing | |
| US11036608B2 (en) | Identifying differences in resource usage across different versions of a software application | |
| CN106909563B (en) | Distributed system | |
| KR20150063233A (en) | Method of processing log data, and system operating the same | |
| CN108460068B (en) | Method, device, storage medium and terminal for importing and exporting report | |
| WO2023197864A1 (en) | Method and device for generating calling topological graph | |
| WO2021238902A1 (en) | Data import method and apparatus, service platform, and storage medium | |
| CN110943872A (en) | Data acquisition service configuration/performance analysis method/system, medium, and server | |
| CN111813666B (en) | Memory leakage positioning method, device, medium and electronic equipment | |
| CN111176951A (en) | Log output configuration/log processing method/system, medium, and server | |
| CN113760242A (en) | Data processing method, device, server and medium | |
| WO2020119551A1 (en) | Log file-based service performance analysis method and apparatus, and electronic device | |
| CN115391215B (en) | Method and application for full link debugging under micro-service architecture | |
| CN104572638A (en) | Data reading and writing method and device | |
| CN116501700A (en) | APP formatted file offline storage method, device, equipment and storage medium | |
| US9659041B2 (en) | Model for capturing audit trail data with reduced probability of loss of critical data | |
| CN112860412B (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN115237889A (en) | Database switching method and device, storage medium and computer equipment | |
| US9037815B2 (en) | Analyzing and reducing dump data | |
| US20240127148A1 (en) | Delta based task analysis for ci systems | |
| CN109086293B (en) | Hive file reading and writing method and device | |
| US20230100758A1 (en) | Unique identifier creation and management for elaborated platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200519 |