CN114697205A - Log processing method and device - Google Patents

Log processing method and device Download PDF

Info

Publication number
CN114697205A
CN114697205A CN202210367785.0A CN202210367785A CN114697205A CN 114697205 A CN114697205 A CN 114697205A CN 202210367785 A CN202210367785 A CN 202210367785A CN 114697205 A CN114697205 A CN 114697205A
Authority
CN
China
Prior art keywords
log
protocol
target
application
analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210367785.0A
Other languages
Chinese (zh)
Other versions
CN114697205B (en
Inventor
潘晓皖
蔡伟
黄卫涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Bilibili Technology Co Ltd
Original Assignee
Shanghai Bilibili Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Bilibili Technology Co Ltd filed Critical Shanghai Bilibili Technology Co Ltd
Priority to CN202210367785.0A priority Critical patent/CN114697205B/en
Publication of CN114697205A publication Critical patent/CN114697205A/en
Application granted granted Critical
Publication of CN114697205B publication Critical patent/CN114697205B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06Management of faults, events, alarms or notifications
    • H04L41/069Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/04Protocols for data compression, e.g. ROHC
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/06Notations for structuring of protocol data, e.g. abstract syntax notation one [ASN.1]
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Debugging And Monitoring (AREA)
  • Computer And Data Communications (AREA)

Abstract

The application provides a log processing method and a log processing device, wherein the log processing method comprises the following steps: acquiring application service information associated with a service end using a target communication protocol; determining an application log and a protocol log corresponding to the target communication protocol according to the application service information; creating a target log file associated with the service terminal according to the application log and the protocol log, and writing the target log file into a log storage space; under the condition that a log analysis request is received, reading the target log file in the log storage space and uploading the target log file to a log analysis platform; and displaying a log analysis interface corresponding to the log analysis request through the log analysis platform, wherein the log analysis interface comprises log association information corresponding to the target log file.

Description

Log processing method and device
Technical Field
The present application relates to the field of computer communications technologies, and in particular, to a log processing method and apparatus.
Background
With the development of internet technology, internet transport layer protocols become the basis for communication between clients and servers. The TCP/IP protocol suite underlies the internet, where transport layer protocols include TCP and UDP protocols, which are more lightweight but less reliable than TCP. To solve the problem; QUIC arose and addressed various requirements faced by the transport and application layers, including handling more connections, security, and low latency. The method realizes the support of more service scenes by fusing the characteristics of protocols such as TCP, TLS, HTTP/2 and the like.
Disclosure of Invention
In view of this, the present application provides a log processing method. The application also relates to a log processing device, a computing device and a computer readable storage medium, which are used for solving the problem that the transmission based on the QUIC protocol causes difficulty in analyzing the service content of the transmission layer in the prior art.
According to a first aspect of the embodiments of the present application, there is provided a log processing method applied to a server, including:
acquiring application service information associated with a service end using a target communication protocol;
determining an application log and a protocol log corresponding to the target communication protocol according to the application service information;
creating a target log file associated with the service terminal according to the application log and the protocol log, and writing the target log file into a log storage space;
under the condition that a log analysis request is received, reading the target log file in the log storage space and uploading the target log file to a log analysis platform;
and displaying a log analysis interface corresponding to the log analysis request through the log analysis platform, wherein the log analysis interface comprises log association information corresponding to the target log file.
According to a second aspect of the embodiments of the present application, there is provided a log processing apparatus applied to a server, including:
the information acquisition module is configured to acquire application service information associated with a service end using a target communication protocol;
the log determining module is configured to determine an application log and a protocol log corresponding to the target communication protocol according to the application service information;
a write-in space module configured to create a target log file associated with the service end according to the application log and the protocol log, and write the target log file into a log storage space;
the file reading module is configured to read the target log file in the log storage space and upload the target log file to a log analysis platform under the condition that a log analysis request is received;
and the log analysis module is configured to display a log analysis interface corresponding to the log analysis request through the log analysis platform, wherein the log analysis interface contains log association information corresponding to the target log file.
According to a third aspect of embodiments of the present application, there is provided a computing device comprising a memory, a processor and computer instructions stored on the memory and executable on the processor, the processor implementing the steps of the log processing method when executing the instructions.
According to a fourth aspect of embodiments of the present application, there is provided a computer-readable storage medium storing computer instructions which, when executed by a processor, implement the steps of the log processing method.
In order to facilitate a service party to analyze transmission content associated with a service end using a target communication protocol, the log processing method provided by the application can determine an application log and a protocol log generated during transmission by using the target communication protocol according to application service information after acquiring the application service information associated with the service end using the target communication protocol; then, creating a target log file according to the application log and the protocol log and writing the target log file into a log storage space; when the transmission content needs to be analyzed, a target log file can be read in the space according to the log analysis request and uploaded to a log analysis platform; and finally, displaying a log analysis interface responding to the log analysis request through a log analysis platform, wherein the interface comprises log association information of an associated target log file. The network condition of the target communication protocol can be analyzed in multiple dimensions according to the analysis request from the protocol log and the application log, so that the network analysis flexibility is improved, and a business side can debug the program of the target communication protocol service more pertinently.
Drawings
Fig. 1 is a schematic structural diagram of a log processing method according to an embodiment of the present application;
fig. 2 is a flowchart of a log processing method according to an embodiment of the present application;
FIG. 3 is a diagram illustrating a file management interface in a log processing method according to an embodiment of the present application;
fig. 4 is a schematic diagram of a log statistical analysis interface in a log processing method according to an embodiment of the present application;
fig. 5 is a log time analysis interface in a log processing method according to an embodiment of the present application;
fig. 6 is a schematic diagram of a log congestion analysis interface in a log processing method according to an embodiment of the present application;
fig. 7 is a schematic diagram of a log transmission analysis interface in a log processing method according to an embodiment of the present application;
fig. 8 is a processing flow chart of a log processing method according to an embodiment of the present application;
fig. 9 is a schematic structural diagram of a log processing apparatus according to an embodiment of the present application;
fig. 10 is a block diagram of a computing device according to an embodiment of the present application.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. This application is capable of implementation in many different ways than those herein set forth and of similar import by those skilled in the art without departing from the spirit of this application and is therefore not limited to the specific implementations disclosed below.
The terminology used in the one or more embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the present application. As used in one or more embodiments of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present application refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It should be understood that, although the terms first, second, etc. may be used herein in one or more embodiments to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first can also be referred to as a second and, similarly, a second can also be referred to as a first without departing from the scope of one or more embodiments of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
First, the noun terms to which one or more embodiments of the present application relate are explained.
QUIC: (Quick UDP Internet Connection) a UDP-based low-latency Internet transport layer protocol. QUIC addresses well the various requirements faced by today's transport and application layers, including handling more connections, security, and low latency. QUIC incorporates features of protocols including TCP, TLS, HTTP/2, etc., but is based on UDP transport. One of the main goals of QUIC is to reduce connection delay, and when a client first connects to a server, QUIC requires only 1RTT (Round-Trip Time) of delay to establish a reliable and secure connection, which is faster than 1-3 RTTs of TCP + TLS. The client may then locally cache the encrypted authentication information and may implement a 0-RTT connection establishment delay when establishing a connection with the server again. QUIC multiplexes the Multiplexing function (Multiplexing) of HTTP/2 protocol at the same time, but avoids the Head-of-Line Blocking (Head-of-Line Blocking) problem of HTTP/2 because QUIC is based on UDP.
Qlog: the name is Quic Log. Is a log for recording the transmission process of the QUIC. The logs were generated following the specification of the qlog schema.
qvis is known as quicvis and is a QUIC transmission visualization tool. Qlog files, which are transferred by importing a certain number of times into a qvis page, can present the transfer process in a visual form.
TCP: (Transmission Control Protocol) is a connection-oriented, reliable transport layer communication Protocol based on a byte stream, defined by RFC 793 of the IETF.
Qlog schema: a specification of a format for standardizing quic logs is described. Log files in this format may facilitate debug and visual analysis. Qlog may be output in json, csv or protobuf.
deburg: and (5) eliminating the fault of the computer.
In the present application, a log processing method is provided, and the present application relates to a log processing apparatus, a computing device, and a computer-readable storage medium, which are described in detail in the following embodiments one by one.
In practical application, QUIC, as a UDP-based low-latency Internet transport layer protocol, has many advantages compared with the traditional HTTP protocol of TCP. In recent years, as the standardization of QUIC advances, more and more websites and applications start to use the QUIC protocol, but debug and analysis tools for QUIC network transmission are relatively deficient. Although the prior art can analyze the QUIC network transmission through qvis, the functional details are not perfect, and when analyzing Qlog, the Qlog is in accordance with the general Qlog schema standard, so that no support is provided for undefined parts of the standard. Qlog files have different contents in different implementations, and therefore an effective solution is needed to solve the above problems.
Referring to the schematic diagram shown in fig. 1, in order to facilitate a service party to analyze transmission content associated with a service end using a target communication protocol, the log processing method provided in the present application may determine, according to application service information, an application log and a protocol log generated when the service end using the target communication protocol is transmitted, after obtaining the application service information associated with the service end using the target communication protocol; then, creating a target log file according to the application log and the protocol log and writing the target log file into a log storage space; when the transmission content needs to be analyzed, a target log file can be read in the space according to the log analysis request and uploaded to a log analysis platform; and finally, displaying a log analysis interface responding to the log analysis request through a log analysis platform, wherein the interface comprises log association information of an associated target log file. The network condition of the target communication protocol can be analyzed in multiple dimensions according to the analysis request from the protocol log and the application log, so that the network analysis flexibility is improved, and a business side can debug the program of the target communication protocol service more pertinently.
Fig. 2 is a flowchart illustrating a log processing method according to an embodiment of the present application, where the method is applied to a server, and specifically includes the following steps:
step S202, obtaining the application service information related to the service end using the target communication protocol.
Specifically, the server is a server capable of analyzing based on a target log file of an associated target communication protocol, and belongs to a server to which a service end of a maintenance application program or a website belongs, that is, a log for analyzing the associated application program or the website, and the log is associated with the target communication protocol. Correspondingly, the target communication protocol is specifically referred to as the QUIC protocol. Correspondingly, the service end specifically refers to a client and a server which use a QUIC protocol for communication, and the client is provided with an application program or a website corresponding to the service end, and the application program or the website is used for communication with the server by using the QUIC protocol. Correspondingly, the application service specifically refers to recording information of a mutual corresponding relationship between the application log and the protocol log, and the application log and the protocol log having the corresponding relationship can be acquired through the information, so that subsequent log analysis and processing operations are facilitated.
It should be noted that the application service information is information generated by combining a communication transmission process between the client and the server according to an application log and a protocol log about an application program generated by the client and the server in a communication process between the client and the server.
That is to say, before the client and the server perform communication transmission, it needs to ensure that the client and the server perform communication using the target communication protocol, and in order to perform accurate analysis on the communication process corresponding to the target communication protocol in the analysis stage, the server needs to issue related parameters associated with the target communication protocol to the client and the server, respectively, so as to ensure that the generation operation of the log related to the target communication protocol can be completed in the transmission stage. In this embodiment, the specific implementation manner is as follows:
determining a client and a server using the target communication protocol; sending protocol parameters and protocol log generation information corresponding to the target communication protocol to the client and the server; and in the process that the client and the server interact by utilizing the target communication protocol, determining the application service information according to the protocol parameters and the protocol log generation information.
Specifically, the protocol parameter refers to a protocol parameter associated with a target communication protocol issued by the server after it is determined that the client and the server need to use the target communication protocol for communication, and includes, but is not limited to, an enabled congestion algorithm, a connection keep-alive default duration, a maximum number of concurrent requests, and the like, and is used for establishing a transmission channel for the client and the server to use the target communication protocol for transmission according to the parameter. Correspondingly, the protocol log generation information specifically refers to information informing the client and the server to generate a log corresponding to the associated target communication protocol, and includes but is not limited to a log generation determination condition, a log sampling rate, a log storage path, and the like, and is used for collecting logs according to the information in the communication stage.
Based on this, in order to support the subsequent analysis of the network transmission process using the target communication protocol, so as to adjust the communication connection between the client and the server, and improve the network transmission efficiency, firstly, it is required to ensure that the client and the server use the target communication protocol for network transmission, and secondly, the server issues the protocol parameters and the protocol log generation information corresponding to the target communication protocol to the client and the server, so that the client and the server can perform network transmission according to the protocol parameters and the protocol log generation information, and the transmission process conforms to the standard of the target communication protocol. Finally, in the process that the client and the server interact by using the target communication protocol, the application service information of the associated service end can be determined according to the protocol parameters and the protocol log generation information, so that the subsequent operation of determining the log according to the application service information and storing the log can be performed conveniently. Thereby supporting log analysis phase usage.
That is to say, when the client and the server perform network communication, the client and the server interact with each other according to the protocol parameters and the protocol log generation information, and the interaction process follows the target communication protocol, so that when the client and the server interact with each other, the client and the server can successfully acquire the related log according to the protocol log generation information to support the log analysis and processing operation.
It should be noted that the interaction between the client and the server is completed by depending on the application installed in the client, that is, the user submits a relevant request to the server by using the application installed in the client, and the server feeds back information to the client according to the relevant request, thereby completing network transmission. For example, an application installed at a client is a player, when a user watches a video at the client using the player, the user sends a video playing request to a server, the server feeds back a video segment corresponding to the video playing request to the client according to the video playing request, so as to play the video segment on the player of the client, and the interaction in the process needs to follow the protocol parameters of a target communication protocol, namely, a congestion algorithm used during network congestion, a connection keep-alive default time length, a maximum concurrent request number and the like, which all need to communicate according to the parameters; when the relevant log is collected, the log generation is completed according to the protocol log generation information, that is, the log generation judgment condition, the log sampling rate, the log storage path and the like are all required to perform the log generation processing according to the information.
In summary, the server issues the protocol parameters and the protocol log generation information to the client and the server, so that the client and the server can perform network transmission according to the target communication protocol, and in the process, the application service information is created according to the protocol parameters and the protocol log generation information, thereby facilitating subsequent log analysis processing on the basis.
Further, in the process of interaction between the client and the server, in order to ensure that the generated logs can be used in the analysis stage, the logs can be collected and then sent to the service log retrieval system for management, so that the service log retrieval system can be requested to obtain related logs for analysis and processing operations when the logs are used at any time. In this embodiment, the specific implementation manner is as follows:
creating an application log and a protocol log corresponding to the target application according to the protocol parameters and the protocol log generation information; uploading the application log and the protocol log to a service log retrieval system, and determining the application service information according to an uploading result; the service log retrieval system stores the application log and the protocol log into a local storage space, or sends the application log and the protocol log to a storage terminal for storage.
Specifically, the application log refers to a log for recording related operations of the application installed on the client, and includes, but is not limited to, a log for recording user operations received by the application, a log for requesting the server through the application, a log for receiving/sending requests by the application, and the like. I.e., logs related to associated network transmissions, logs related to application execution, logs related to application updates, etc. Correspondingly, the protocol log specifically refers to a related log for recording a related target communication protocol, including but not limited to a packet loss log, a delay log, a bandwidth log, and the like. I.e. logs related to the associated network transmissions. The method is used for recording the operation related to the whole transmission process in the network transmission. Correspondingly, the service log retrieval system specifically refers to a system for managing the protocol log and the application log, and is deployed at a server to which the application program belongs, that is, has a communication relationship with the server. Correspondingly, the local storage space specifically refers to a local corresponding storage space for running the log processing service; correspondingly, the storage end specifically refers to another end which is in communication with the server and is dedicated to storing the log. Correspondingly, the target application specifically refers to an application installed on the client and running on the client, and the network communication with the server is performed through the application.
Based on the above, in the process of interaction between the client and the server by using the target communication protocol, the application log corresponding to the target application and the protocol log associated with network transmission can be created according to the protocol parameters and the protocol log generation information issued to the client and the server. Because the generation of the protocol log depends on the interaction of the target application server, and the application log is a log generated in the running of the target application, the protocol log and the application log have a corresponding relation in time, and the protocol log and the application log can be uploaded to a service log retrieval system for managing the associated target communication protocol according to the corresponding relation, so that the application service information corresponding to the service end is determined according to the log uploading result, and the log management and analysis can be conveniently carried out according to the application service information in the follow-up process.
Furthermore, after the service log retrieval system receives the application log and the protocol log of the associated target application, in order to facilitate the subsequent analysis and management stage and rapidly complete the log reading processing operation, the service log retrieval system can store the application log in a local storage space so as to achieve the purpose that the application stage can be rapidly read and processed; or the application log and the protocol log are stored in the storage end, so that the cost of maintaining the storage space of the service log retrieval system is reduced, and more computing resources can be called to perform log management operation.
In specific implementation, in order to support downstream service analysis after log generation, both a client and a server can be ensured to use a QUIC protocol, and the server can determine protocol parameters related to the used QUIC protocol and protocol log generation information according to issued configuration, namely information of a Qlog log is generated; the QUIC application server is used as an execution main body under a log management mechanism, and the log management operation is realized through the butt joint server and the service retrieval system. Before storage, in order to achieve the purpose of saving storage space, the protocol log can be compressed, and finally, the compressed protocol log and application log are stored in a local storage space or uploaded to a storage end for storage.
In addition, the client can also generate additional other logs according to the requirements for assisting the server in positioning other problems, so that the purpose of reasonably debugging the transmission channel is achieved.
In conclusion, the protocol logs and the application logs are stored in the local storage space or the storage end according to the requirements, so that the storage resources can be reasonably utilized, and the subsequent adjustment of network transmission is more convenient on the basis.
Step S204, determining an application log and a protocol log corresponding to the target communication protocol according to the application service information.
Specifically, after the application service information associated with the service end is received, it is further described that the application log and the protocol log associated with the service end are collected at this time, and the application log and the protocol log are stored in the specified positions; in order to facilitate the log analysis and processing operation of the downstream service, after the log collection is completed, the server can manage the log, namely, report, index and store the collected log. Therefore, uploading and analysis can be completed quickly based on the operation result after log management conveniently in the analysis stage, that is, the application log with the clear associated target generated by the service end and the protocol log transmitted by the associated network can be read directly based on the application service information. To facilitate management at this stage.
Furthermore, considering that the protocol log and the application log are managed by the service log retrieval system before being processed, when it is determined that the log needs to be reported, indexed and stored, a read request needs to be sent to the service log retrieval system, and a management operation needs to be performed on the log fed back by the request. In this embodiment, the specific implementation manner is as follows:
determining a log corresponding relation according to the application service information, and sending a log reading request carrying the log corresponding relation to the service log retrieval system; and receiving the application log corresponding to the target application and the protocol log corresponding to the target communication protocol, which are fed back by the service log retrieval system according to the log corresponding relation, as a response of the log reading request.
Specifically, the log correspondence relationship is a correspondence relationship between the protocol log and the application log, and the correspondence relationship is used for representing a relationship between the protocol log and the application log in a time dimension and is used for embodying network transmission content which will occur when a target application runs a target transaction at a certain time; correspondingly, the log reading request specifically refers to reading a protocol log and an application log associated with the client and the server within a certain time period.
Based on this, after receiving the application service information associated with the service end, it is indicated that the log needs to be reported, indexed and stored according to the application service information. So as to be convenient for the log analysis stage to use; therefore, the log corresponding relation between the protocol log and the application log can be determined according to the application service information, the log reading request is created by combining the log corresponding relation, the log reading request carrying the log corresponding relation is sent to the service log retrieval system, the service log retrieval system determines the protocol log and the application log which need to be managed according to the log corresponding relation in the log reading request and transmits the protocol log and the application log back to the server, and the server can be used for subsequent management processing operation after receiving the application log and the protocol log which are fed back by the service log retrieval system according to the log reading request.
In conclusion, the application service information is combined to request the service log retrieval system for the application log and the protocol log, so that the requested logs can be ensured to be more accurate, and the service log retrieval system is independently used for maintaining before management, so that the efficiency of a management stage can be effectively improved, and the downstream service can be conveniently used.
Step S206, a target log file associated with the service terminal is created according to the application log and the protocol log, and the target log file is written into a log storage space.
Specifically, after the application log and the protocol log are obtained, the obtained application log and the protocol log are further managed, and in order to facilitate the use of the downstream service, a target log file associated with the service end may be created based on the protocol log and the application log, and then written into a log storage space, so as to implement management operation on the log.
The target log file is a file which is obtained by packaging an application log and a protocol date which have an association relation and is used as a binding relation to be stored in a log storage space, so that the target log file can be directly read when a downstream service is used conveniently to obtain the application log and the protocol log which have the association relation; correspondingly, the log storage space specifically refers to a storage space used by the server for storing log files, and the log files stored in the storage space may be used in the analysis stage for debugging the network transmission channel.
Furthermore, when the application log and the protocol log are managed, the application log is used for recording brief information of each request of the client, such as url of the request, content of request resources, request initiation time and completion time, node name of the request to the server, response status code of the request and the like, wherein one request corresponds to one log, considering that the protocol log and the content recorded in the application log correspond to different objects; the content of the protocol log record is more detailed, for example, the change condition of the protocol stack in the connection period between the client and the server is recorded, and the record can be understood as the packet capturing information of the QUIC protocol. The analysis phase therefore requires the use of protocol logs and/or application logs; before this, in order to ensure that the analysis stage can be completed quickly, the analysis stage and the management stage may be subjected to binding management, and in this embodiment, the specific implementation manner is as in step S2062 to step S2064.
Step S2062, determining the communication identifier corresponding to the service end according to the target communication protocol, and creating the target log file associated with the service end according to the application log and the protocol log.
Specifically, the communication identifier specifically refers to a connection id corresponding to the client and the server when performing communication interaction. And the communication identification has uniqueness. Based on this, in consideration of the fact that the log analysis processing operation needs to be performed in a targeted manner in the log analysis stage, in the log management process, the communication identifier corresponding to the service end can be determined according to the target communication protocol, and meanwhile, the target log file is created according to the application log and the protocol log, so that the target log file can be stored according to the communication identifier in the following process.
Further, in the log management stage, in order to save the storage space, the volume of the log may be reduced by compressing twice, in this embodiment, the specific implementation manner is as follows:
compressing the protocol log to obtain a target protocol log; creating an initial log file according to the target protocol log and the application log; compressing the initial log file to obtain the target log file associated with the service end; the application log records log features corresponding to the protocol log, and the log storage space updates the storage space capacity according to a preset release strategy.
Specifically, the target protocol log refers to a log obtained by compressing the protocol log; correspondingly, the initial log file specifically refers to a log file integrating a target protocol log and an application log; correspondingly, the log characteristics specifically refer to characteristics of a corresponding protocol log recorded in the application log, and the protocol log mainly records a packet capturing record of the QUIC protocol, so that the application log needs to carry information such as a connection ID of the QUIC, packet loss, retransmission, byte number of receiving and transmitting and the like in a reporting stage, and a developer can perform basic judgment on a network condition through the characteristics of the protocol log when retrieving on a service log retrieval platform. Correspondingly, the release policy specifically refers to a policy for performing space management on a log storage space, and the policy may be a time management policy, that is, earlier logs may be cleared according to a time sequence at set intervals, or may also be other policies, for example, a log with normally closed connections is cleared, and a log with abnormal connections is reserved, that is, the release policy only needs to keep a log that needs to be analyzed and processed, and in actual application, the release policy may be set according to an actual application scenario, and this embodiment is not limited herein.
Based on this, in order to reduce the consumption of the storage space, after the protocol log is obtained, the protocol log is firstly compressed for one time to obtain a target protocol log; then, packaging the application log having the association relation with the protocol log and the target protocol log to obtain an initial log file; and then, carrying out secondary compression on the initial log file so as to obtain a target log file corresponding to the initial log file according to a compression result, wherein the target log file is used for subsequent storage processing.
In addition, in order to facilitate normal processing of the log in the reporting stage, log features corresponding to the protocol log need to be recorded in the application log. Meanwhile, a release strategy needs to be set for the log storage space, so that after the log storage space is loaded, space release processing operation can be performed according to the strategy, and it is ensured that a newly obtained target log file can be written into the log storage space.
In summary, the target log file is obtained by adopting a twice compression mode, so that the consumption of the file on space resources can be further reduced, more log files can be stored in an effective space, and the retention time of the log files is prolonged.
Step S2064, configuring the communication identifier for the target log file, and writing the target log file into the log storage space according to the communication identifier.
Specifically, after the communication identifier and the target log file are determined, the target log file can be written into the log storage space according to the communication identifier, so that the unique corresponding relation between the communication identifier and the target log file is established, and after log management is completed, the required target log file can be read through the communication identifier in an analysis stage and used for analysis processing.
That is, after obtaining the application log and the protocol log, the target log file cid.qlog may be named by the connection ID, and the protocol log may perform a field compression once inside the program, generate an initial log file, perform a secondary compression, and then place the initial log file in a log storage space, where the log storage space may also update the size of the space according to a preset release policy, so as to ensure that the disk space is available.
Meanwhile, in order to ensure smooth reporting, log features (qlog features) of the protocol logs are written in Application logs (Application logs) and reported together, so that a user can conveniently retrieve and position storage positions of the protocol logs to perform subsequent analysis and processing operations after extraction.
In summary, by storing the target log file by using the communication identifier, the unique corresponding relationship between the identifier and the file can be established in the reporting and storing stages, so that the position of the protocol log can be quickly located in the analysis and processing stage, the analysis and processing operation can be completed, and the log analysis and processing efficiency can be improved.
And step S208, under the condition that a log analysis request is received, reading the target log file in the log storage space and uploading the target log file to a log analysis platform.
Specifically, after the log management is completed, further, when a log analysis request is received, it is described that the network transmission channel needs to be analyzed according to the collected log at this time, so as to perform debugging according to an analysis result, thereby improving the network transmission performance. Based on this, at this moment, the target log file is read in the log storage space according to the log analysis request and uploaded to the log analysis platform, so that the transmission performance is analyzed through the log analysis platform, the visualization of parameters can be realized through the log analysis platform, and the use of developers is facilitated.
The log analysis request specifically refers to a request for analyzing a target log file, and is used for determining network transmission performance; the log analysis request includes, but is not limited to, a log statistical analysis request, a log time analysis request, a log congestion analysis request, a log transmission analysis request, and the like, and is used for reading different logs for performing visualization processing operations according to different requests. Correspondingly, the log analysis platform specifically refers to a platform capable of converting the log into a visual interface according to the target log file and the log analysis request.
Furthermore, when log analysis processing is performed, a large number of log files are stored in the log storage space, so that after a log analysis request is obtained, a target log file can be read according to identification information recorded in the request for analysis processing operation; in this embodiment, the specific implementation manner is as follows:
under the condition that the log analysis request is received, determining log identification information according to the log analysis request; and reading the log storage space according to the log identification information, determining the target log file according to the reading result and uploading the target log file to the log analysis platform.
Specifically, the log identifier specifically refers to a unique identifier corresponding to the target log file, and is used for reading the target log file in the log storage space. Based on this, under the condition of receiving the log analysis request, the log analysis request can be analyzed at this time to obtain a log identifier; and then, reading operation is carried out in the log storage space according to the log identification so as to obtain a target log file according to the reading result, and the target log file is sent to the log analysis platform, so that subsequent visualization processing is facilitated.
It should be noted that, in order to ensure that the target log file can be accurately obtained in the reading process, the log identifier may record the relevant contents of the associated protocol log and the target log, and is used to specify the log file to which the log to be analyzed belongs, thereby ensuring the reading accuracy.
In conclusion, the target log file is read on the basis of the log identification, so that the reading accuracy can be effectively guaranteed, visual processing operation can be performed in response to the log analysis request during analysis processing, and a developer can more conveniently debug a network transmission channel.
Step S210, displaying, by the log analysis platform, a log analysis interface corresponding to the log analysis request, where the log analysis interface includes log association information corresponding to the target log file.
Specifically, after the target log file is uploaded to the log analysis platform, the log analysis platform can respond to the log analysis request and generate a log analysis interface according to the log association information corresponding to the target log file. The log analysis interface is specifically an interface created according to log association information in the target log file, and is used for performing visualization processing on the log content and creating a log analysis interface corresponding to the log analysis request. The log related information is used for representing log contents which need to be visualized in the log analysis request.
In other words, in the log analysis stage, the application log and the protocol log in the target log file can be pulled on the log analysis platform, automatic analysis is performed according to the log analysis request, and finally a log analysis interface containing log related information is created according to the analysis result, so that convenience is brought to analysis of developers.
Further, when the target log file is uploaded to the log analysis platform, considering that the target log file may exist in different forms, such as being stored in a local storage space or being stored in a storage end, a plurality of file import interfaces may be provided to the developer on the file management interface, referring to the file management interface diagram shown in fig. 3, a network file import interface (corresponding to the storage end), a local file import interface (corresponding to the local storage space), a standard file import interface, a large file import interface, and the like are provided to the developer, so that the developer can conveniently upload the target log file according to needs.
Furthermore, all parameters related to network transmission are recorded in the log, so that a developer can submit different analysis requests according to different requirements during analysis, and the log analysis platform can respond to different analysis requests to output different log analysis interfaces.
(1) Under the condition that the log analysis request is a log statistical analysis request, extracting the protocol log from the target log file; and creating the log associated information according to the communication protocol information recorded in the protocol log, and displaying a log statistical analysis interface containing the log associated information through the log analysis platform.
Specifically, the log statistical analysis request specifically refers to a protocol log statistical data analysis request submitted for a target log file; correspondingly, the communication protocol information specifically refers to information associated when network transmission is recorded, including but not limited to connection duration, version information corresponding to the communication protocol, packet loss information, delay information, and the like. Correspondingly, the log statistical analysis interface is specifically an interface created based on the communication protocol information, and is used for visualizing the information and facilitating browsing of developers.
Based on this, under the condition that the developer uploads the log statistical analysis request, it is described that the developer needs to check the communication protocol information about the target communication protocol, at this time, the protocol log can be read in the target log file, and the log related information is created according to the communication protocol information recorded in the protocol log, so that the log statistical analysis interface containing the log related information is displayed through the log analysis platform.
Referring to the log statistical analysis interface shown in fig. 4, after a developer uploads a target log file, according to qlog logs in the target log file, it is determined that the connection duration corresponding to network transmission is 66.921284s, and the QUIC VERSION information is QUIC _ VERSION _ IETF _ draw _ 29; the connection of 0-RTT is false, meanwhile, the progress bar shows that the ratio of the total packet sending number to the total packet loss number is 71.32%, and the ratio of the total packet sending byte statistics to the total packet loss byte statistics is 70.96%; url information is {000cf85 … … 68u }, etc.
(2) Under the condition that the log analysis request is a log time analysis request, extracting the protocol log and the application log from the target log file; creating the log associated information according to the protocol log, a target event recorded in the application log and time information corresponding to the target event; and displaying a log time analysis interface containing the log association information through the log analysis platform.
Specifically, the log time analysis request specifically refers to a time sequence analysis request submitted for a target log file; correspondingly, the target event specifically refers to a transmission event generated in the communication process, and the transmission event corresponds to transmission time, namely time information between the client and the server; correspondingly, the log time analysis interface is specifically an interface created based on the time information, and is used for visualizing the time sequence information, so that a developer can conveniently browse the time sequence information.
Based on this, under the condition that the developer uploads the log time analysis request, it is described that the developer needs to check the time sequence information about the target communication protocol, at this time, the application log and the protocol log can be extracted from the target log file, and the log association information is created according to the events recorded in the application log and the protocol log and the corresponding time information thereof, so that the log time analysis interface containing the log association information is displayed through the log analysis platform.
Referring to the log time analysis interface shown in fig. 5 (a), the log time analysis interface includes a communication schematic diagram corresponding to the client and the server, each time line corresponding to each segment has a time point corresponding to an event, each time point is an event point, and the schematic diagram shown in fig. 5 (b) can be generated by clicking the event point, so as to show an event time offset, a packet number, and a packet size corresponding to the event; meanwhile, in the log time analysis interface, each packet can be previewed, and in order to facilitate global browsing of developers, intervals with higher time intervals can be displayed in a zooming mode, so that the developers can check time sequence information more conveniently.
(3) Under the condition that the log analysis request is a log congestion analysis request, extracting the protocol log and the application log from the target log file; creating the log associated information according to the application log and the request operation information recorded in the protocol log; and displaying a log congestion analysis interface containing the log association information through the log analysis platform.
Specifically, the log congestion analysis request specifically refers to a network transmission congestion analysis request submitted for a target log file; correspondingly, the request operation information specifically refers to operation information corresponding to the congestion condition, and is used for representing information such as network congestion condition, congestion relief time and the like; correspondingly, the log congestion analysis interface is specifically an interface created based on the request operation information, and is used for visualizing the congestion information, so that a developer can browse the congestion information conveniently.
Based on this, under the condition that a developer uploads log congestion analysis, it is described that the developer needs to check congestion information about a target communication protocol, at this time, an application log and a protocol log may be extracted from a target log file, and log association information is created according to operation request information recorded in the application log and the protocol log, so that a log congestion analysis interface including the log association information is displayed through a log analysis platform.
Referring to the log congestion analysis interface shown in fig. 6, congestion statistical information can be generated according to a log congestion analysis request submitted by a developer, and for the convenience of the developer to perform analysis, the congestion statistical information can be obtained by providing a control for emptying all canvases, a control for drawing all information, a control for restoring coordinate axes, a control for selecting a horizontal region and a rectangular region, a control for identifying an estimation sending rate, a congestion information display switch control, a control for amplifying RTT information of the next canvas, a BBR mode display switch control, and the like.
That is, in order to assist the developer in analyzing the congestion information through the log analysis platform, a distribution diagram of the received acknowledgement (ack), sent acknowledgement (send), and lost (lost) data according to time and accumulated byte number, a congestion control Window size (congestion Window), an in-flight data amount (inflight bytes), and a network delay data (rtt) may be displayed to the developer for plotting. Meanwhile, the statistics and the drawing are carried out on the related index data respectively collected by several congestion control algorithms used by QUIC, and the statistical chart is drawn according to the BBR congestion control algorithm with reference to fig. 6. For reflecting the transitions of the different state machines in the lifetime of the QUIC connection, and the parameters (BDP, Bandwidth, RTT) on which the state transitions depend.
For example, a player of the QUIC service is installed in the client, and a play log corresponding to the player may be generated to know when the client player initiates a request, what kind of request is initiated, a behavior operation (stop) of the player, a change in a buffer status, whether a card is generated, and so on. The log of the player and the qlog of the server are presented together, and the association of both ends can be mined out for troubleshooting problems and assisting performance tuning.
(4) Under the condition that the log analysis request is a log transmission analysis request, extracting the protocol log and the application log from the target log file; creating the log associated information according to the transmission information recorded in the application log and the protocol log; and displaying a log transmission analysis interface containing the log association information through the log analysis platform.
Specifically, the log transmission analysis request specifically refers to a multiplexing analysis request submitted for a target log file; correspondingly, the transmission information specifically refers to information corresponding to each transport stream; for characterizing the multiplexing condition. Correspondingly, the log transmission analysis interface is specifically an interface created based on transmission information, and is used for visualizing multiplexing information, so that a developer can browse the information conveniently.
Based on this, when the developer uploads the log transmission analysis request, which indicates that the developer needs to view the multiplexing information about the target communication protocol, the application log and the protocol log may be extracted from the target log file, and the log association information may be created according to the transmission information recorded in the application log and the protocol log, so as to display the log transmission analysis interface including the log association information through the log analysis platform.
Referring to the log transmission analysis interface shown in fig. 7, multiplexing analyzes all streams connected by quic frame by frame, and after introducing qlog files and clicking corresponding controls, the streams can be analyzed and arranged in sequence according to a time axis, so that the phenomenon positioning problem can be conveniently observed, and meanwhile, the service information and the statistical information on each stream are displayed, so that developers can view data (representing retransmission data, and can presume a network condition and calculate an effective load) and multiplexing conditions overlapped on each stream, and the service content which may cause problems can be presumed from the service information (HTTP request content) on the corresponding time axis.
In the log processing method provided by the application, in order to facilitate a service party to analyze transmission content associated with a service end using a target communication protocol, after application service information associated with the service end using the target communication protocol is acquired, an application log and a protocol log generated during transmission by using the target communication protocol are determined according to the application service information; then, creating a target log file according to the application log and the protocol log and writing the target log file into a log storage space; when the transmission content needs to be analyzed, a target log file can be read in the space according to the log analysis request and uploaded to a log analysis platform; and finally, displaying a log analysis interface responding to the log analysis request through a log analysis platform, wherein the interface comprises log association information of an associated target log file. The network condition of the target communication protocol can be analyzed in multiple dimensions according to the analysis request from the protocol log and the application log, so that the network analysis flexibility is improved, and a business side can debug the program of the target communication protocol service more pertinently.
The following description will further describe the log processing method in conjunction with fig. 8 by taking an application of the log processing method provided in the present application in a QUIC protocol communication scenario as an example. Fig. 8 shows a processing flow chart of a log processing method provided in an embodiment of the present application, which specifically includes the following steps:
step S802, determining a client and a server using a target communication protocol.
Step S804, the protocol parameters and the protocol log generation information corresponding to the target communication protocol are issued to the client and the server.
Step S806, in the process of the client and the server interacting with the target communication protocol, creating an application log and a protocol log corresponding to the target application according to the protocol parameter and the protocol log generation information.
Step S808, uploading the application log to a log retrieval system, and determining application service information according to an uploading result, wherein the application service information record includes a field of a corresponding storage location of the protocol log in the application log.
Step S810, acquiring application service information associated with a service end using a target communication protocol.
Step S812, determining the log mapping relationship according to the application service information, and sending a log reading request carrying the log mapping relationship to the service log retrieval system.
Step S814, receiving an application log corresponding to the target application and a protocol log corresponding to the target communication protocol, which are fed back by the service log retrieval system according to the log correspondence, as a response to the log reading request.
Step S816, determining a communication identifier corresponding to the service end according to the target communication protocol.
Step S818, compress the protocol log to obtain the target protocol log.
Step S820, configuring a communication identifier for the target protocol log, and writing the target protocol log into a log storage space according to the communication identifier.
In step S822, in the case of receiving the log analysis request, the log identification information is determined according to the log analysis request.
Step S824, reading the log storage space according to the log identification information, determining the target protocol log according to the reading result, and uploading the target protocol log to the log analysis platform.
Step S826, a log analysis interface corresponding to the log analysis request is displayed through the log analysis platform, where the log analysis interface includes log association information corresponding to the target protocol log.
In summary, in order to facilitate the service party to analyze the transmission content associated with the service end using the target communication protocol, after acquiring the application service information associated with the service end using the target communication protocol, an application log and a protocol log generated during transmission using the target communication protocol may be determined according to the application service information; then, creating a target log file according to the application log and the protocol log and writing the target log file into a log storage space; when the transmission content needs to be analyzed, a target log file can be read in the space according to the log analysis request and uploaded to a log analysis platform; and finally, displaying a log analysis interface responding to the log analysis request through a log analysis platform, wherein the interface comprises log association information of an associated target log file. The network condition of the target communication protocol can be analyzed in multiple dimensions according to the analysis request from the protocol log and the application log, so that the network analysis flexibility is improved, and a business side can debug the program of the target communication protocol service more pertinently.
Corresponding to the above method embodiment, the present application further provides an embodiment of a log processing apparatus, and fig. 9 shows a schematic structural diagram of the log processing apparatus provided in an embodiment of the present application. As shown in fig. 9, the apparatus includes:
an obtaining information module 902 configured to obtain application service information associated with a service end using a target communication protocol;
a determining log module 904 configured to determine, according to the application service information, an application log and a protocol log corresponding to the target communication protocol;
a write space module 906 configured to create a target log file associated with the service end according to the application log and the protocol log, and write the target log file into a log storage space;
a file reading module 908 configured to, in a case that a log analysis request is received, read the target log file in the log storage space and upload the target log file to a log analysis platform;
a log analysis module 910 configured to display a log analysis interface corresponding to the log analysis request through the log analysis platform, where the log analysis interface includes log association information corresponding to the target log file.
In an optional embodiment, the log processing apparatus further includes:
a service information determining module configured to determine a client and a server using the target communication protocol; sending protocol parameters and protocol log generation information corresponding to the target communication protocol to the client and the server; and in the process that the client and the server interact by utilizing the target communication protocol, determining the application service information according to the protocol parameters and the protocol log generation information.
In an optional embodiment, the obtain information module 902 is further configured to:
creating an application log and a protocol log corresponding to the target application according to the protocol parameters and the protocol log generation information; uploading the application log and the protocol log to a service log retrieval system, and determining the application service information according to an uploading result; the service log retrieval system stores the application log and the protocol log into a local storage space, or sends the application log and the protocol log to a storage terminal for storage.
In an optional embodiment, the obtain information module 902 is further configured to:
determining a log corresponding relation according to the application service information, and sending a log reading request carrying the log corresponding relation to the service log retrieval system; and receiving the application log corresponding to the target application and the protocol log corresponding to the target communication protocol, which are fed back by the service log retrieval system according to the log corresponding relation, as a response of the log reading request.
In an alternative embodiment, the write space module 906 is further configured to:
determining a communication identifier corresponding to the service end according to the target communication protocol, and creating the target log file associated with the service end according to the application log and the protocol log; and configuring the communication identifier for the target log file, and writing the target log file into the log storage space according to the communication identifier.
In an alternative embodiment, the write space module 906 is further configured to:
compressing the protocol log to obtain a target protocol log; creating an initial log file according to the target protocol log and the application log; compressing the initial log file to obtain the target log file associated with the service end; the application log records log features corresponding to the protocol log, and the log storage space updates the storage space capacity according to a preset release strategy.
In an optional embodiment, the read file module 908 is further configured to:
under the condition that the log analysis request is received, determining log identification information according to the log analysis request; and reading the log storage space according to the log identification information, determining the target log file according to the reading result and uploading the target log file to the log analysis platform.
In an optional embodiment, the log analysis module 910 is further configured to:
under the condition that the log analysis request is a log statistical analysis request, extracting the protocol log from the target log file; and creating the log associated information according to the communication protocol information recorded in the protocol log, and displaying a log statistical analysis interface containing the log associated information through the log analysis platform.
In an optional embodiment, the log analysis module 910 is further configured to:
under the condition that the log analysis request is a log time analysis request, extracting the protocol log and the application log from the target log file; creating the log associated information according to the protocol log, a target event recorded in the application log and time information corresponding to the target event; and displaying a log time analysis interface containing the log association information through the log analysis platform.
In an optional embodiment, the log analysis module 910 is further configured to:
under the condition that the log analysis request is a log congestion analysis request, extracting the protocol log and the application log from the target log file; creating the log associated information according to the application log and the request operation information recorded in the protocol log; and displaying a log congestion analysis interface containing the log association information through the log analysis platform.
In an optional embodiment, the log analysis module 910 is further configured to:
under the condition that the log analysis request is a log transmission analysis request, extracting the protocol log and the application log from the target log file; creating the log associated information according to the transmission information recorded in the application log and the protocol log; and displaying a log transmission analysis interface containing the log association information through the log analysis platform.
In order to facilitate a service party to analyze transmission content associated with a service end using a target communication protocol, the log processing device provided by the application can determine an application log and a protocol log generated during transmission by using the target communication protocol according to application service information after acquiring the application service information associated with the service end using the target communication protocol; then, creating a target log file according to the application log and the protocol log and writing the target log file into a log storage space; when the transmission content needs to be analyzed, a target log file can be read in the space according to the log analysis request and uploaded to a log analysis platform; and finally, displaying a log analysis interface responding to the log analysis request through a log analysis platform, wherein the interface comprises log association information of an associated target log file. The network condition of the target communication protocol can be analyzed in multiple dimensions according to the analysis request from the protocol log and the application log, so that the network analysis flexibility is improved, and a business side can debug the program of the target communication protocol service more pertinently.
The foregoing is a schematic configuration of a log processing apparatus of the present embodiment. It should be noted that the technical solution of the log processing apparatus belongs to the same concept as the technical solution of the log processing method described above, and details that are not described in detail in the technical solution of the log processing apparatus can be referred to the description of the technical solution of the log processing method described above.
Fig. 10 shows a block diagram of a computing device 1000 according to an embodiment of the present application. The components of the computing device 1000 include, but are not limited to, memory 1010 and a processor 1020. The processor 1020 is coupled to the memory 1010 via a bus 1030, and the database 1050 is used to store data.
Computing device 1000 also includes access device 1040, access device 1040 enabling computing device 1000 to communicate via one or more networks 1060. Examples of such networks include the Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the internet. Access device 1040 may include one or more of any type of network interface (e.g., a Network Interface Card (NIC)) whether wired or wireless, such as an IEEE802.11 Wireless Local Area Network (WLAN) wireless interface, a worldwide interoperability for microwave access (Wi-MAX) interface, an ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a bluetooth interface, a Near Field Communication (NFC) interface, and so forth.
In one embodiment of the present application, the above-described components of computing device 1000 and other components not shown in FIG. 10 may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device architecture shown in FIG. 10 is for purposes of example only and is not limiting as to the scope of the present application. Those skilled in the art may add or replace other components as desired.
Computing device 1000 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), a mobile phone (e.g., smartphone), a wearable computing device (e.g., smartwatch, smartglasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. Computing device 1000 may also be a mobile or stationary server.
Wherein, the processor 1020 implements the steps of the log processing method when executing the instructions.
The above is an illustrative scheme of a computing device of the present embodiment. It should be noted that the technical solution of the computing device and the technical solution of the log processing method belong to the same concept, and details that are not described in detail in the technical solution of the computing device can be referred to the description of the technical solution of the log processing method.
An embodiment of the present application further provides a computer readable storage medium, which stores computer instructions, and when the instructions are executed by a processor, the instructions implement the steps of the log processing method as described above.
The above is an illustrative scheme of a computer-readable storage medium of the present embodiment. It should be noted that the technical solution of the storage medium belongs to the same concept as the technical solution of the log processing method, and details that are not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the log processing method.
The foregoing description has been directed to specific embodiments of this application. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.
The computer instructions comprise computer program code which may be in the form of source code, object code, an executable file or some intermediate form, or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.
It should be noted that, for the sake of simplicity, the above-mentioned method embodiments are described as a series of acts or combinations, but those skilled in the art should understand that the present application is not limited by the described order of acts, as some steps may be performed in other orders or simultaneously according to the present application. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required in this application.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
The preferred embodiments of the present application disclosed above are intended only to aid in the explanation of the application. Alternative embodiments are not exhaustive and do not limit the invention to the precise embodiments described. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the application and its practical applications, to thereby enable others skilled in the art to best understand and utilize the application. The application is limited only by the claims and their full scope and equivalents.

Claims (14)

1. A log processing method is applied to a server and comprises the following steps:
acquiring application service information associated with a service end using a target communication protocol;
determining an application log and a protocol log corresponding to the target communication protocol according to the application service information;
creating a target log file associated with the service terminal according to the application log and the protocol log, and writing the target log file into a log storage space;
under the condition that a log analysis request is received, reading the target log file in the log storage space and uploading the target log file to a log analysis platform;
and displaying a log analysis interface corresponding to the log analysis request through the log analysis platform, wherein the log analysis interface comprises log association information corresponding to the target log file.
2. The method of claim 1, wherein before the step of obtaining the application service information associated with the service end using the target communication protocol is executed, the method further comprises:
determining a client and a server using the target communication protocol;
sending protocol parameters and protocol log generation information corresponding to the target communication protocol to the client and the server;
and in the process that the client and the server interact by utilizing the target communication protocol, determining the application service information according to the protocol parameters and the protocol log generation information.
3. The method of claim 2, wherein determining the application service information according to the protocol parameters and the protocol log generation information comprises:
creating an application log and a protocol log corresponding to the target application according to the protocol parameters and the protocol log generation information;
uploading the application log and the protocol log to a service log retrieval system, and determining the application service information according to an uploading result;
the service log retrieval system stores the application log and the protocol log into a local storage space, or sends the application log and the protocol log to a storage terminal for storage.
4. The method of claim 3, wherein the determining the application log and the protocol log corresponding to the target communication protocol according to the application service information comprises:
determining a log corresponding relation according to the application service information, and sending a log reading request carrying the log corresponding relation to the service log retrieval system;
and receiving the application log corresponding to the target application and the protocol log corresponding to the target communication protocol, which are fed back by the service log retrieval system according to the log corresponding relation, as a response of the log reading request.
5. The method of claim 1, wherein the creating a target log file associated with the service end according to the application log and the protocol log and writing the target log file into a log storage space comprises:
determining a communication identifier corresponding to the service end according to the target communication protocol, and creating the target log file associated with the service end according to the application log and the protocol log;
and configuring the communication identifier for the target log file, and writing the target log file into the log storage space according to the communication identifier.
6. The method of claim 5, wherein the creating the target log file associated with the service end according to the application log and the protocol log comprises:
compressing the protocol log to obtain a target protocol log;
creating an initial log file according to the target protocol log and the application log;
compressing the initial log file to obtain the target log file associated with the service end;
the application log records log features corresponding to the protocol log, and the log storage space updates the storage space capacity according to a preset release strategy.
7. The method of claim 1, wherein reading the target log file in the log storage space and uploading to a log analysis platform upon receiving a log analysis request comprises:
under the condition that the log analysis request is received, determining log identification information according to the log analysis request;
and reading the log storage space according to the log identification information, determining the target log file according to the reading result and uploading the target log file to the log analysis platform.
8. The method according to any one of claims 1 to 7, wherein the displaying, by the log analysis platform, a log analysis interface corresponding to the log analysis request includes:
under the condition that the log analysis request is a log statistical analysis request, extracting the protocol log from the target log file;
and creating the log associated information according to the communication protocol information recorded in the protocol log, and displaying a log statistical analysis interface containing the log associated information through the log analysis platform.
9. The method according to any one of claims 1 to 7, wherein the displaying, by the log analysis platform, a log analysis interface corresponding to the log analysis request includes:
under the condition that the log analysis request is a log time analysis request, extracting the protocol log and the application log from the target log file;
creating the log associated information according to the protocol log, a target event recorded in the application log and time information corresponding to the target event;
and displaying a log time analysis interface containing the log association information through the log analysis platform.
10. The method according to any one of claims 1 to 7, wherein the displaying, by the log analysis platform, a log analysis interface corresponding to the log analysis request includes:
under the condition that the log analysis request is a log congestion analysis request, extracting the protocol log and the application log from the target log file;
creating the log associated information according to the application log and the request operation information recorded in the protocol log;
and displaying a log congestion analysis interface containing the log association information through the log analysis platform.
11. The method according to any one of claims 1 to 7, wherein the displaying, by the log analysis platform, a log analysis interface corresponding to the log analysis request includes:
under the condition that the log analysis request is a log transmission analysis request, extracting the protocol log and the application log from the target log file;
creating the log associated information according to the transmission information recorded in the application log and the protocol log;
and displaying a log transmission analysis interface containing the log association information through the log analysis platform.
12. A log processing device applied to a server includes:
the information acquisition module is configured to acquire application service information associated with a service end using a target communication protocol;
the log determining module is configured to determine an application log and a protocol log corresponding to the target communication protocol according to the application service information;
a write-in space module configured to create a target log file associated with the service end according to the application log and the protocol log, and write the target log file into a log storage space;
the file reading module is configured to read the target log file in the log storage space and upload the target log file to a log analysis platform under the condition that a log analysis request is received;
and the log analysis module is configured to display a log analysis interface corresponding to the log analysis request through the log analysis platform, wherein the log analysis interface comprises log association information corresponding to the target log file.
13. A computing device comprising a memory, a processor, and computer instructions stored on the memory and executable on the processor, wherein the processor implements the steps of the method of any one of claims 1-11 when executing the instructions.
14. A computer-readable storage medium storing computer instructions, which when executed by a processor, perform the steps of the method of any one of claims 1 to 11.
CN202210367785.0A 2022-04-08 2022-04-08 Log processing method and device Active CN114697205B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210367785.0A CN114697205B (en) 2022-04-08 2022-04-08 Log processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210367785.0A CN114697205B (en) 2022-04-08 2022-04-08 Log processing method and device

Publications (2)

Publication Number Publication Date
CN114697205A true CN114697205A (en) 2022-07-01
CN114697205B CN114697205B (en) 2024-03-12

Family

ID=82142433

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210367785.0A Active CN114697205B (en) 2022-04-08 2022-04-08 Log processing method and device

Country Status (1)

Country Link
CN (1) CN114697205B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115102930A (en) * 2022-06-28 2022-09-23 海南乾唐视联信息技术有限公司 Conference map generation method and device, electronic equipment and readable storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20020097291A (en) * 2001-06-20 2002-12-31 (주)엔토시스 Method for analyzing log of wireless internet
US20070150253A1 (en) * 2005-12-28 2007-06-28 Microsoft Corporation Reviewing effectiveness of communication rules system
CN106603296A (en) * 2016-12-20 2017-04-26 北京奇虎科技有限公司 Log processing method and device
CN108512694A (en) * 2018-03-05 2018-09-07 北京信安世纪科技股份有限公司 A kind of method and device of server log analysis
CN111209170A (en) * 2019-10-21 2020-05-29 江苏卓易信息科技股份有限公司 Rule-based user behavior log acquisition and analysis method and system
CN112887388A (en) * 2021-01-20 2021-06-01 每日互动股份有限公司 Data processing system based on sandbox environment

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20020097291A (en) * 2001-06-20 2002-12-31 (주)엔토시스 Method for analyzing log of wireless internet
US20070150253A1 (en) * 2005-12-28 2007-06-28 Microsoft Corporation Reviewing effectiveness of communication rules system
CN106603296A (en) * 2016-12-20 2017-04-26 北京奇虎科技有限公司 Log processing method and device
CN108512694A (en) * 2018-03-05 2018-09-07 北京信安世纪科技股份有限公司 A kind of method and device of server log analysis
CN111209170A (en) * 2019-10-21 2020-05-29 江苏卓易信息科技股份有限公司 Rule-based user behavior log acquisition and analysis method and system
CN112887388A (en) * 2021-01-20 2021-06-01 每日互动股份有限公司 Data processing system based on sandbox environment

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115102930A (en) * 2022-06-28 2022-09-23 海南乾唐视联信息技术有限公司 Conference map generation method and device, electronic equipment and readable storage medium

Also Published As

Publication number Publication date
CN114697205B (en) 2024-03-12

Similar Documents

Publication Publication Date Title
CN108400909B (en) Traffic statistical method, device, terminal equipment and storage medium
US20200228433A1 (en) Computer-readable recording medium including monitoring program, programmable device, and monitoring method
US9781224B2 (en) Content transmitting system, method for optimizing network traffic in the system, central control device and local caching device
CN105024872B (en) The method and device of applied in network performance test
US9210090B1 (en) Efficient storage and flexible retrieval of full packets captured from network traffic
US8341285B2 (en) Method and system for transferring files
WO2021164261A1 (en) Method for testing cloud network device, and storage medium and computer device
US10616378B2 (en) Adaptive session intelligence extender
JP6451359B2 (en) Communication recording apparatus, communication recording system, communication recording program, and communication recording method
CN114697205B (en) Log processing method and device
CN105450479A (en) Method and apparatus for monitoring module
CN108880875A (en) The transmission method and device of log message
CN115499230A (en) Network attack detection method and device, equipment and storage medium
RU2668394C2 (en) Packet collection method and system, network device and network management centre
US11991061B2 (en) Processing local area network diagnostic data
US8230002B2 (en) Method and system for automatic setup in web-based applications
KR102423039B1 (en) Real-time packet data storing method and apparatus for mass network monitoring
US20140143264A1 (en) Policy event driven remote desktop recording across a data network
CN114281258A (en) Service processing method, device, equipment and medium based on data storage
KR102423038B1 (en) Real-time packet data collection method and apparatus for mass network monitoring
CN113438503A (en) Video file restoration method and device, computer equipment and storage medium
CN113285904A (en) RDP-based method for analyzing disk mapping file information
CN114189565B (en) Head area restoration system, method and related equipment
CN106664223A (en) Detection method and detection device for the number of shared access hosts
US20230421651A1 (en) On demand end user monitoring for automated help desk support

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant