CN111339051B - Log processing method, system, device and storage medium - Google Patents
Log processing method, system, device and storage medium Download PDFInfo
- Publication number
- CN111339051B CN111339051B CN202010117194.9A CN202010117194A CN111339051B CN 111339051 B CN111339051 B CN 111339051B CN 202010117194 A CN202010117194 A CN 202010117194A CN 111339051 B CN111339051 B CN 111339051B
- Authority
- CN
- China
- Prior art keywords
- log
- logs
- request information
- communication module
- global
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a log processing method, a system, equipment and a storage medium, wherein the method comprises the following steps: establishing a structural body of a log file to be generated, and respectively sending request information to a source application and a target application; responding to the request information, and respectively generating a plurality of first logs corresponding to the source application and a plurality of second logs corresponding to the target application according to the structural body of the log file to be generated; establishing a distributed transmission information body; according to the global log ID, respectively determining second logs which are the same as the global log ID contained in the first logs for all the first logs, and combining the first logs and the second logs to form a log group; transmitting the log group to a second communication module according to the distributed transmission information body; and comparing the first log with the second log by using the second communication module to determine the position of the error in the second log. The method and the device realize the full-link tracking service process, are favorable for accurately positioning the error log link, and reduce the software maintenance cost.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, a system, a device, and a storage medium for processing a log.
Background
Nowadays of networking, the traditional business and new business are gradually increased and changed day by day, and the diversification of various industries is promoted, so that the market mechanisms of most industries compete fiercely particularly on differentiated, refined and vertical services. For example, in the important vertical fields of e-commerce and the like, such as e-commerce, payment, freight transportation and other services, market feedback also clusters a plurality of innovative products such as increase of channel parties and value increase of experience. This indirectly considers the complexity and robustness of the system-wide and ultimately is a comprehensive measure of the efficiency of the system operation. Therefore, tracking, observing and monitoring of service execution links and operation records of a vertical system or a platform are particularly important, and tiny detail tracking is a basic support for improving operation efficiency and product experience.
The core problems encountered by most vertical systems today include: increasingly complex distributed service systems, a wide variety of middleware, frameworks, distributed cache calls, and call relationships that are difficult to clarify. In particular, with respect to, for example, complicated call links, time and effort consuming problem location, after-knowledge awareness of urgent problems, observation capabilities of service sampling, which service requests have been processed by which modules? Which modules are hotspots? How the runtime dependencies between modules are, etc. So many and complex problems reflect how better the link tracking and service performance of the system can be gathered. The prior art can not realize the full-link tracking of the business process, which can not assist an analyst to observe the business process from a log view angle when a problem occurs, can not quickly and accurately position the problem log, and can not solve the problem in time. The simple understanding is that the service level logs lack full link dotting marks and the like, and the full-flow concatenation of the service level logs cannot be achieved. Finally, how much the complex business does, what it does, the reason for doing the same, and the like are unclear.
Disclosure of Invention
Aiming at the problems in the prior art, the invention aims to provide a log processing method, a system, equipment and a storage medium, which solve the problems that the prior art cannot realize a full link tracking service process and cannot realize accurate positioning of an error log link.
In order to achieve the above object, the present invention provides a log processing method, including the steps of:
s10, establishing a structural body of a log file to be generated, and respectively sending request information to a source application and a target application;
s20, responding to the request information, and respectively generating a plurality of first logs corresponding to the source application and a plurality of second logs corresponding to the target application according to the structural body of the log file to be generated, wherein each first log and each second log comprise global log IDs, and the global log IDs correspond to the request information one by one;
s30, establishing a distributed transmission information body, wherein the distributed transmission information body is used for realizing communication between two communication modules adopting different communication protocols;
s40, according to the global log ID, respectively determining the second logs which are the same as the global log ID contained in the first logs for all the first logs, and combining the first logs and the second logs to form a log group;
s50, transmitting the log group to a second communication module according to the distributed transmission information body;
s60, comparing the first log and the second log in the log group by using the second communication module, and determining the position of an error in the second log.
Preferably, the sending request information to the source application and the target application respectively comprises:
sending request information to a source application;
the source application responds to the request information and generates a head log containing the request information;
sending the header log to a second communication module;
and the second communication module extracts the request information from the head log and sends the request information to a target application.
Preferably, the step S40 includes:
s401, acquiring additional information required by comparison, wherein the additional information comprises a current application role and a current comparison stage;
s402, according to the global log ID, respectively determining the second logs with the same global log ID as the global log ID contained in the first logs for all the first logs, and combining the first logs and the second logs to form a log group;
the step S50 includes:
and transmitting the first log, the second log and the additional information to the second communication module according to the distributed transmission information body, and processing a data stream in a transmission process based on a message middleware.
Preferably, the step S50 includes: and converting the data stream in the transmission process into a byte stream based on the message middleware, and sending the byte stream to the second communication module.
Preferably, the step S60 includes:
storing the first log and the second log by using the second communication module;
and the second communication module takes out the first log at regular time and compares the first log with the second log.
Preferably, the structural body of the log file to be generated further includes a log version number, a node type, a local address, a thread number, a service code, a message body, and a timestamp.
Preferably, the second communication module is a server.
The invention also provides a log processing system, which comprises:
the log file structure body building module is used for building a structure body of a log file to be generated and respectively sending request information to the source application and the target application;
a log file generation module, configured to respond to the request information, and generate, according to the structure of the log file to be generated, a plurality of first logs corresponding to the source application and a plurality of second logs corresponding to the target application, where each of the first logs and each of the second logs include a global log ID, and the global log IDs correspond to the request information one to one;
the distributed transmission information body establishing module is used for establishing a distributed transmission information body which is used for realizing communication between two communication modules adopting different communication protocols;
a log group determination module, configured to determine, for all the first logs, the second logs that are the same as the global log IDs included in the first logs according to the global log IDs, and combine the first logs and the second logs to form a log group;
the log file transmission module is used for transmitting the log group to the second communication module according to the distributed transmission information body;
and the target application error positioning module is used for comparing the first log with the second log in the log group by using the second communication module to determine the position of an error in the second log.
The present invention also provides a log processing apparatus, including:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of any of the log processing methods described above via execution of the executable instructions.
The present invention also provides a computer-readable storage medium storing a program which, when executed, implements the steps of any of the log processing methods described above.
Compared with the prior art, the invention has the following advantages and prominent effects:
according to the log processing method, the system, the device and the storage medium, the global log IDs are respectively arranged in the first log generated by the source application and the second log generated by the target application, and the global log IDs correspond to the request information one by one, so that the first log and the second log can be compared under the same request information; the method and the device realize the tracking of the execution condition of the service link and are beneficial to better analyzing the execution condition of the service. On the other hand, by the log comparison realized by the method, software developers can be helped to position the error log link more quickly and accurately, the software maintenance cost is reduced, and the development efficiency is improved. Meanwhile, the method is beneficial to combing the flow and the incidence relation of the depth service, achieves the effect of quickly extracting the key element information in the context, improves the product operation efficiency and ensures the service stability.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, with reference to the accompanying drawings.
FIG. 1 is a schematic flowchart of a log processing method according to an embodiment of the present invention;
FIG. 2 is a schematic view of a detailed flow chart of step S10 in FIG. 1;
FIG. 3 is a schematic structural diagram of a log processing system according to an embodiment of the present disclosure;
FIG. 4 is a schematic structural diagram of a log processing apparatus according to an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and thus their repetitive description will be omitted.
As shown in fig. 1, the embodiment of the present invention discloses a log processing method, which includes the following steps:
and S10, establishing a structural body of the log file to be generated, and respectively sending request information to the source application and the target application. Specifically, the structure of the log file to be generated further includes elements such as a log version number, a node type, a local address, a thread number, a service code, a message body, and a timestamp. These elements describe the link information of the log, the module described by the log, the type of transaction, the time of occurrence, the content of the occurrence, etc. The message body also has its own structure information in order to improve the clarity, normalization, readability, and the like of the clear log body. In this embodiment, the message body at least includes the following field information:
target: describing the object name of the current request information, and identifying the target in operation, such as a transaction serial number;
content: the specific operation content of the current request information is referred, and the things which are being done are described, such as 100 yuan is paid;
cause: the reason for the current request, describing the cause of the operation, such as the driver being on-line to pay a subscription;
evolutionPath: the current path of realizing describes what kind of route and accomplishes current operation, if through the equity calculation, the driver benefits 10 yuan, only needs to pay 90 yuan, through such specification, the collection and the accomodating of the information of integrating more easily.
As shown in fig. 2, in this embodiment, the sending request information to the source application and the target application respectively includes:
s101, sending request information to a source application;
s102, the source application responds to the request information and generates a head log containing the request information;
s103, sending the head log to a second communication module;
s104, the second communication module extracts the request information from the head log and sends the request information to a target application.
Specifically, after the source application generates the head log, the head log is sent to the second communication module, i.e., the server. And after receiving the head log, the server identifies the comparison start mark, namely, the server judges that the log comparison work of the source application and the target application is started. And then the server extracts the request information from the head log and sends the request information to the target application. The request information sent to the source application and the request information sent to the target application are multiple.
And S20, responding to the request information, and respectively generating a plurality of first logs corresponding to the source application and a plurality of second logs corresponding to the target application according to the structure of the log file to be generated, wherein each first log and each second log comprise global log IDs, and the global log IDs correspond to the request information one by one. Specifically, each source application is attributed to a service line and each target application is also attributed to a service line. The source application prints a first log with a plurality of pieces of log information according to a request message. The global log ID contained within the first log helps the first log to match the request information sent to the source application. For the target application, a second log printed for a request message also has a plurality of log information. And the global log ID contained in the second log also corresponds one-to-one to the request information sent to the target application. Therefore, the first log and the second log responding to the same request information can be grouped during log comparison, and subsequent log comparison work is facilitated.
In the generation process of the first log and the second log, this embodiment needs to be completed by using a pre-established log component. The log component comprises log printing, log context, log section and log source. The log printing is used for loading a printing suite with mature bottom layers of various programming languages, and provides basic output log methods, including printing methods corresponding to various levels such as INFO, WARN, ERROR, DEBUG and the like, and supporting different parameter types or extensive types and the like. The log context carries key information of the log link, including module type, node type, service code, native address, and message ID. The four pieces of key information can be associated with the tracking information of each service log and information concatenation of each child node in a full link. For specific tools or module types, such as a class of services, public toolkits and the like, a log section can be adopted to finish the unified log printing of all sub-functions, and common log section printing forms comprise front-section printing and back-section printing. The log source is used for packaging the log standard definitions into corresponding structural bodies of the log files to be generated through the grammars of the programming languages.
And S30, establishing a distributed transmission information body, wherein the distributed transmission information body is used for realizing communication between two communication modules adopting different communication protocols. The distributed transport information body converts a data stream transmitted between two communication modules into a byte stream based on socket communication and tcp (Transmission Control Protocol) Protocol, for example, the distributed transport information body receives a data stream sent by one communication module (which may be a source application or a target application in this embodiment), converts the data stream into a byte stream and stores the byte stream, and then sends the byte stream to another communication module (which may be a server in this embodiment). Therefore, both communication parties do not need to concern about the storage of the transmission link and only need to finish the uploading of the service log. And the two communication parties read the client side packaged by the distributed transmission information body and directly complete information pushing and information pulling.
And S40, respectively identifying the second logs which are the same as the global log ID contained in the first logs for all the first logs according to the global log IDs, and combining the first logs and the second logs to form a log group. Specifically, since the step S20 has implemented the one-to-one correspondence of the request information with the first log generated by the source application by the global log ID, and the one-to-one correspondence of the request information with the second log generated by the target application by the global log ID. Then, according to the global log ID, the corresponding second log matched with each first log can be determined, and then the first log and the second log obtained by matching are combined to be sent to the server as a log group for comparison.
As a preferred embodiment, the step S40 includes:
s401, acquiring additional information required by comparison, wherein the additional information comprises a current application role and a current comparison stage.
S402, identifying the second log having the same global log ID as the global log ID included in the first log for all the first logs based on the global log ID, and combining the first log and the second log to form a log group.
The current application role is one of a source application and a target application. Log alignment is not done at once, so there are multiple alignment stages. The generated first log and the second log also have a plurality of pieces of log information, and the server compares one piece of log information in the first log with one piece of log information in the second log in each comparison stage. The current comparison stage represents the comparison stage currently performed by the server.
And S50, transmitting the log group to a second communication module according to the distributed transmission information body.
Specifically, the step S50 includes:
and transmitting the first log, the second log and the additional information to the second communication module according to the distributed transmission information body, and processing a data stream in a transmission process based on a message middleware. In this embodiment, the message middleware is equivalent to a storage intermediate medium, so that a data stream transmitted by one communication party can be temporarily stored in the transmission process, and the other communication party can be used at any time. Therefore, the situation that one communication party can transmit the next batch of data streams only after waiting for the other communication party to acquire the data is avoided, and the data transmission efficiency is improved.
In another embodiment of the present application, the step S50 includes: and converting the data stream in the transmission process into a byte stream based on the message middleware, and sending the byte stream to the second communication module. Therefore, data transmission can be facilitated, and the safety of data transmission is ensured.
It should be noted that, in this embodiment, the second communication modules are all servers.
And S60, comparing the first log and the second log in the log group by using the second communication module, and determining the position of an error in the second log.
As a preferred embodiment of the present invention, the step S60 includes:
storing the first log and the second log by using the second communication module;
the second communication module takes out the first log at regular time and compares the first log with the second log. The second communication module plays a role in temporarily storing the first log and the second log, then the log information is taken out at a regular period for comparison, and when the second log is in a place different from the first log, the second communication module, namely the server, judges the second log to be an error position.
As shown in fig. 3, an embodiment of the present invention further discloses a log processing system 3, which includes:
the log file structure establishing module 31 is configured to establish a structure of a log file to be generated, and send request information to the source application and the target application respectively;
a log file generating module 32, configured to respond to the request information, and generate, according to the structure of the log file to be generated, a plurality of first logs corresponding to the source application and a plurality of second logs corresponding to the target application, where each of the first logs and each of the second logs includes a global log ID, and the global log IDs correspond to the request information one to one;
a distributed transmission information body establishing module 33, configured to establish a distributed transmission information body, where the distributed transmission information body is used to implement communication between two communication modules that use different communication protocols;
a log group identification module 34 configured to identify, for each of the first logs, the second log having the same global log ID as the global log ID included in the first log according to the global log ID, and combine the first log and the second log to form a log group;
a log file transmission module 35, configured to transmit the log group to a second communication module according to the distributed transmission information body;
and a target application error positioning module 36, configured to compare the first log with the second log in the log group by using the second communication module, and determine a location of an error in the second log.
In this embodiment, the log file structure establishing module 31 includes:
a source application request sending unit, configured to send request information to a source application;
a head log generating unit, configured to control the source application to respond to the request information and generate a head log including the request information;
the head log sending unit is used for sending the head log to a second communication module;
and the target application request sending unit is used for controlling the second communication module to extract the request information from the head log and sending the request information to the target application.
It is understood that the log processing system of the present invention also includes other existing functional modules that support the operation of the log processing system. The log processing system shown in fig. 3 is only an example, and should not bring any limitation to the function and the range of use of the embodiment of the present invention.
The log processing system in this embodiment is used to implement the log processing method, so for the specific implementation steps of the log processing system, reference may be made to the description of the log processing method, and details are not described here again.
The embodiment of the invention also discloses log processing equipment, which comprises a processor and a memory, wherein the memory stores executable instructions of the processor; the processor is configured to perform the steps of the log processing method described above via execution of executable instructions. Fig. 4 is a schematic structural diagram of a log processing device disclosed by the invention. An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 4. The electronic device 600 shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 4, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different platform components (including the memory unit 620 and the processing unit 610), a display unit 640, etc.
Wherein the storage unit stores program code which can be executed by the processing unit 610 such that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above-mentioned log processing method section of the present specification. For example, processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM) 6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include programs/utilities 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination thereof may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with the other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, to name a few.
The invention also discloses a computer readable storage medium for storing a program, which when executed implements the steps in the log processing method. In some possible embodiments, the various aspects of the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the invention described in the above-mentioned log processing methods of this specification, when the program product is run on the terminal device.
As described above, the program of the computer-readable storage medium of this embodiment, when executed, by providing the global log IDs in the first log generated by the source application and the second log generated by the destination application, respectively, can implement comparison of the first log and the second log in the same request information because the global log IDs and the request information are in one-to-one correspondence; the method realizes the tracking of the system link execution condition. On the other hand, by comparing the logs, the method and the system can help software developers to quickly and accurately locate the log link with errors, reduce the software maintenance cost and improve the development efficiency. Meanwhile, the method is beneficial to combing the flow and the association relation of the depth service, achieves the effect of quickly extracting the key element information in the context, improves the product operation efficiency and ensures the service stability.
Fig. 5 is a schematic structural diagram of a computer-readable storage medium of the present invention. Referring to fig. 5, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
According to the log processing method, the system, the device and the storage medium provided by the embodiment of the invention, the global log IDs are respectively arranged in the first log generated by the source application and the second log generated by the target application, and the global log IDs are in one-to-one correspondence with the request information, so that the comparison of the first log and the second log under the same request information can be realized; the method realizes the tracking of the system link execution condition. On the other hand, by the log comparison realized by the method, software developers can be helped to position the error log link more quickly and accurately, the software maintenance cost is reduced, and the development efficiency is improved. Meanwhile, the method is beneficial to combing the flow and the incidence relation of the depth service, achieves the effect of quickly extracting the key element information in the context, improves the product operation efficiency and ensures the service stability.
The foregoing is a further detailed description of the invention in connection with specific preferred embodiments and it is not intended to limit the invention to the specific embodiments described. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (9)
1. A log processing method, comprising the steps of:
s10, establishing a structural body of a log file to be generated, and respectively sending request information to a source application and a target application;
s20, responding to the request information, and respectively generating a plurality of first logs corresponding to the source application and a plurality of second logs corresponding to the target application according to the structural body of the log file to be generated, wherein each first log and each second log comprise global log IDs, and the global log IDs correspond to the request information one by one;
s30, establishing a distributed transmission information body, wherein the distributed transmission information body is used for realizing communication between two communication modules adopting different communication protocols;
s40, according to the global log ID, respectively determining the second logs which are the same as the global log ID contained in the first logs for all the first logs, and combining the first logs and the second logs to form a log group;
s50, transmitting the log group to a second communication module according to the distributed transmission information body;
s60, comparing the first log with the second log in the log group by using the second communication module, and determining the position of an error in the second log;
the sending request information to the source application and the target application respectively comprises:
sending request information to a source application;
the source application responds to the request information and generates a head log containing the request information;
sending the header log to a second communication module;
and the second communication module extracts the request information from the head log and sends the request information to a target application.
2. The log processing method according to claim 1, wherein the step S40 includes:
s401, acquiring additional information required by comparison, wherein the additional information comprises a current application role and a current comparison stage;
s402, according to the global log ID, respectively determining the second logs with the same global log ID as the global log ID contained in the first logs for all the first logs, and combining the first logs and the second logs to form a log group;
the step S50 includes:
and transmitting the first log, the second log and the additional information to the second communication module according to the distributed transmission information body, and processing a data stream in a transmission process based on a message middleware.
3. The log processing method according to claim 2, wherein the step S50 includes: and converting the data stream in the transmission process into a byte stream based on the message middleware, and sending the byte stream to the second communication module.
4. The log processing method according to claim 1, wherein the step S60 includes:
storing the first log and the second log by using the second communication module;
and the second communication module takes out the first log at regular time and compares the first log with the second log.
5. The log processing method according to claim 1, wherein the structure of the log file to be generated further includes a log version number, a node type, a native address, a thread number, a service code, a message body, and a timestamp.
6. The log processing method of claim 1, wherein the second communication module is a server.
7. A log processing system, the system comprising:
the log file structure body building module is used for building a structure body of a log file to be generated and respectively sending request information to the source application and the target application;
a log file generation module, configured to respond to the request information, and generate, according to the structure of the log file to be generated, a plurality of first logs corresponding to the source application and a plurality of second logs corresponding to the target application, where each of the first logs and each of the second logs include a global log ID, and the global log IDs correspond to the request information one to one;
the distributed transmission information body establishing module is used for establishing a distributed transmission information body, and the distributed transmission information body is used for realizing communication between two communication modules adopting different communication protocols;
a log group determination module configured to determine, for all the first logs, the second logs that are the same as the global log IDs included in the first logs, respectively, according to the global log IDs, and combine the first logs and the second logs to form a log group;
the log file transmission module is used for transmitting the log group to the second communication module according to the distributed transmission information body;
the target application error positioning module is used for comparing the first log and the second log in the log group by utilizing the second communication module to determine the position of an error in the second log;
the log file structure body building module is further used for:
sending request information to a source application;
the source application responds to the request information and generates a head log containing the request information;
sending the header log to a second communication module;
and the second communication module extracts the request information from the head log and sends the request information to a target application.
8. A log processing apparatus characterized by comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the log processing method of any one of claims 1 to 6 via execution of the executable instructions.
9. A computer-readable storage medium storing a program, wherein the program is configured to implement the steps of the log processing method according to any one of claims 1 to 6 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010117194.9A CN111339051B (en) | 2020-02-25 | 2020-02-25 | Log processing method, system, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010117194.9A CN111339051B (en) | 2020-02-25 | 2020-02-25 | Log processing method, system, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111339051A CN111339051A (en) | 2020-06-26 |
| CN111339051B true CN111339051B (en) | 2022-10-04 |
Family
ID=71181830
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010117194.9A Active CN111339051B (en) | 2020-02-25 | 2020-02-25 | Log processing method, system, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111339051B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112115023B (en) * | 2020-09-02 | 2022-08-05 | 厦门安胜网络科技有限公司 | Acquisition method and system for data link |
| CN112737869A (en) * | 2021-04-02 | 2021-04-30 | 北京宇信科技集团股份有限公司 | Link tracking method and system of ESB (Enterprise service bus) system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107330034A (en) * | 2017-06-26 | 2017-11-07 | 百度在线网络技术（北京）有限公司 | A kind of log analysis method and device, computer equipment, storage medium |
| CN107766354A (en) * | 2016-08-17 | 2018-03-06 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus for being used to ensure data correctness |
-
2020
- 2020-02-25 CN CN202010117194.9A patent/CN111339051B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107766354A (en) * | 2016-08-17 | 2018-03-06 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus for being used to ensure data correctness |
| CN107330034A (en) * | 2017-06-26 | 2017-11-07 | 百度在线网络技术（北京）有限公司 | A kind of log analysis method and device, computer equipment, storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111339051A (en) | 2020-06-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111083225B (en) | Data processing method and device in Internet of things platform and Internet of things platform | |
| CN110891057B (en) | Protocol conversion method, device, equipment and storage medium | |
| US8239493B2 (en) | Automated server controlled client-side logging | |
| CN107292117B (en) | Processing method and device for quality guarantee of mass shared medical images | |
| CN111339051B (en) | Log processing method, system, device and storage medium | |
| CN111796978B (en) | Interface detection method, device, system, equipment and storage medium | |
| CN112860798B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN114356785B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113127108A (en) | Service request processing method and device, storage medium and electronic equipment | |
| CN113360144A (en) | Auxiliary processing method, device, storage medium and program product for software development | |
| US7925788B2 (en) | Systems and methods for universal protocol for case management systems | |
| US7886277B2 (en) | System and method for error checking of failed I/O open calls | |
| CN116627849B (en) | System test method, device, equipment and storage medium | |
| CN113238739A (en) | Plug-in development and data acquisition method, device, electronic equipment and medium | |
| CN110995538B (en) | Network data acquisition method, device, system, equipment and storage medium | |
| CN113238940B (en) | Interface test result comparison method, device, equipment and storage medium | |
| CN114531289A (en) | System interaction method based on artificial intelligence and related equipment | |
| CN111741046B (en) | Data reporting method, data acquisition method, device, equipment and medium | |
| CN114219643A (en) | Transaction calling method, device, equipment and storage medium | |
| CN110377401B (en) | Transaction request processing method, device, server and storage medium based on IDEA | |
| CN110515849B (en) | Breakpoint debugging method, device, system, equipment and storage medium | |
| CN114489667A (en) | Script generation method and device, electronic equipment and storage medium | |
| CN115421824B (en) | Micro-service execution engine, running method and application method | |
| US8321235B2 (en) | Validating an electronic transaction | |
| CN114500679B (en) | can protocol conversion method, can protocol conversion device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220914 Address after: No.123, Kaifa Avenue, Guiyang Economic and Technological Development Zone, 550000, Guizhou Province Applicant after: GUIYANG HUOCHEBANG TECHNOLOGY CO.,LTD. Address before: 210012 3-5 / F, building 4, 170-1 software Avenue, Yuhuatai District, Nanjing City, Jiangsu Province Applicant before: JIANGSU MANYUN SOFTWARE TECHNOLOGY Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20200626 Assignee: Nanjing Manyun Cold Chain Technology Co.,Ltd. Assignor: GUIYANG HUOCHEBANG TECHNOLOGY CO.,LTD. Contract record no.: X2023310000141 Denomination of invention: Log processing methods, systems, devices, and storage media Granted publication date: 20221004 License type: Common License Record date: 20230802 |
|
| EE01 | Entry into force of recordation of patent licensing contract |