CN110851471A - Distributed log data processing method, device and system - Google Patents
Distributed log data processing method, device and system Download PDFInfo
- Publication number
- CN110851471A CN110851471A CN201810848162.9A CN201810848162A CN110851471A CN 110851471 A CN110851471 A CN 110851471A CN 201810848162 A CN201810848162 A CN 201810848162A CN 110851471 A CN110851471 A CN 110851471A
- Authority
- CN
- China
- Prior art keywords
- log
- service
- data
- compliance check
- compliance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The present disclosure relates to the field of data processing technologies, and in particular, to a distributed log data processing method, a distributed log data processing apparatus, a distributed log data processing system, a storage medium, and a computer system. The method comprises the following steps: receiving a compliance audit log of a microservice container; wherein the compliance audit log includes a service data identifier; merging the associated compliance check logs according to the service data identifier to obtain a merged log; checking the merged log according to a first checking rule for checking the service main logic; and when the checking result of the merged log is abnormal, sending a log extracting instruction to the corresponding micro-service container according to the service data identifier so as to obtain a complete micro-service log corresponding to the compliance check log. The method and the device can realize the storage of the log data under the abnormal condition, and reduce the searching difficulty of the log data.
Description
Technical Field
The present disclosure relates to the field of data processing technologies, and in particular, to a distributed log data processing method, a distributed log data processing apparatus, a distributed log data processing system, a storage medium, and a computer system.
Background
With the maturity of cloud computing technology, most large application systems or services adopt distributed micro-service clusters to split a single application system or service into a plurality of supporting micro-services, so that stable operation of the application system or service is ensured. For example, Docker-based micro-servitization clustering. A service request from a user may contain multiple task segments and involve multiple microservice containers. Meanwhile, log data of the application system or the service is also dispersedly stored in each micro-service container.
However, when an application system or a service is abnormal or wrong, or a user requests a service in a wrong way and needs to query a relevant log, the logs of a plurality of micro-service containers need to be checked, so that the data volume for checking the logs is huge, and the difficulty of log checking is increased. In addition, because the logs of the micro service container are mostly provided with a function of deleting regularly, the logs can be deleted certainly when being inquired, and the logs cannot be searched.
It is to be noted that the information disclosed in the above background section is only for enhancement of understanding of the background of the present disclosure, and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The present disclosure aims to provide a distributed log data processing method, a distributed log data processing system, a storage medium, and a computer system, so as to overcome, at least to some extent, the situation that log data is not easy to find due to limitations and defects of related technologies.
Additional features and advantages of the disclosure will be set forth in the detailed description which follows, or in part will be obvious from the description, or may be learned by practice of the disclosure.
According to a first aspect of the present disclosure, there is provided a distributed log data processing method, including:
receiving a compliance audit log of a microservice container; wherein the compliance audit log includes a service data identifier;
merging the associated compliance check logs according to the service data identifier to obtain a merged log;
checking the merged log according to a first checking rule for checking the service main logic;
and when the checking result of the merged log is abnormal, sending a log extracting instruction to the corresponding micro-service container according to the service data identifier so as to obtain a complete micro-service log corresponding to the compliance check log.
In an exemplary embodiment of the present disclosure, the method further comprises:
and when the merged log data is checked to have no abnormity, deleting the merged log and/or the compliance check log corresponding to the merged log.
In an exemplary embodiment of the present disclosure, the service data identification includes: the micro-service container calling system comprises a service request identifier used for marking service requests and a micro-service calling identifier used for identifying the calling sequence of the micro-service container.
In an exemplary embodiment of the present disclosure, the merging the associated compliance check logs according to the service data identifier to obtain a merged log includes:
acquiring at least one compliance check log according to the service request identifier;
and merging the compliance check logs according to the micro-service calling identifier according to a calling sequence.
In an exemplary embodiment of the present disclosure, when it is determined that the merged log has abnormal data, sending a log extraction instruction to the corresponding microservice container according to the service data identifier so as to obtain a complete microservice log corresponding to the compliance check log includes:
when judging that the merged log has abnormal data, generating a log extraction instruction;
and adding the log extraction instruction into a current period log extraction instruction list so as to send the log extraction instruction to the corresponding micro service container at the moment of a preset period.
In an exemplary embodiment of the present disclosure, the complete microservice log comprises:
a log of runs of the microservice container and a log of compliance checks of the microservice container.
In an exemplary embodiment of the present disclosure, an interactive interface is provided, the method further comprising:
receiving a log data query request;
and extracting a corresponding complete micro service log according to the log data query request.
According to a second aspect of the present disclosure, there is provided a distributed log data processing method including:
checking the current task according to a second detection rule for checking the business logic to generate a current compliance check log;
generating a service data identifier in the current compliance check log to obtain the compliance check log;
and sending the compliance check log to a central log server so that the central log server can check the compliance check log.
In an exemplary embodiment of the present disclosure, the adding a service data identifier to the current compliance audit log to obtain a compliance audit log includes:
extracting current log data from the current compliance check log according to a preset field;
and generating the service data identifier and adding the service data identifier to the current log data to obtain the compliance check log.
In an exemplary embodiment of the present disclosure, the method further comprises:
if a log extraction instruction corresponding to the compliance check log is received, extracting a complete micro service log corresponding to the compliance check log according to the log extraction instruction;
wherein the complete microservice log comprises: a log of runs of microservice containers and a log of compliance checks of the microservice containers.
According to a third aspect of the present disclosure, there is provided a distributed log data processing system, comprising: a central log server, and at least one microservice container;
wherein the central log server is configured to perform the method in the first aspect embodiment;
the microservice container is configured to perform the method of the second aspect embodiment described above.
According to a fourth aspect of the present disclosure, there is provided a distributed log data processing apparatus including:
the log data receiving module is used for receiving a compliance check log of the microservice container; wherein the compliance audit log includes a service data identifier;
the log data merging module is used for merging the associated compliance check logs according to the service data identification to obtain a merged log;
the log data inspection module is used for inspecting the merged log according to a first inspection rule for inspecting the service main logic;
and the complete log data extraction module is used for sending a log extraction instruction to the corresponding micro-service container according to the service data identifier when the checking result of the combined log is abnormal so as to obtain the complete micro-service log corresponding to the compliance check log.
According to a fifth aspect of the present disclosure, there is provided a distributed log data processing apparatus comprising:
the log data generation module is used for checking the current task according to a second detection rule for checking the business logic to generate a current compliance check log;
the identification loading module is used for adding a service data identification to the current compliance check log so as to obtain a compliance check log;
and the log data sending module is used for sending the compliance check log to a central log server so as to be convenient for the central log server to check the compliance check log.
According to a sixth aspect of the present disclosure, there is provided a storage medium having stored thereon a computer program which, when executed by a processor, implements the distributed log data processing method described above.
According to a seventh aspect of the present disclosure, there is provided a computer system comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the distributed log data processing method described above via execution of the executable instructions.
In the distributed log data processing method provided by an embodiment of the present disclosure, accurate positioning and fast troubleshooting of abnormal data are realized by receiving at least one compliance check log from a plurality of micro service containers, merging the compliance check logs to obtain a merged log, and examining the merged log according to a preset rule. Meanwhile, when the combined log is judged to be abnormal, the complete micro-service log corresponding to the micro-service container is actively acquired, so that log data is saved, subsequent processing of the data is facilitated, and the searching difficulty of the log data is reduced.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure. It is to be understood that the drawings in the following description are merely exemplary of the disclosure, and that other drawings may be derived from those drawings by one of ordinary skill in the art without the exercise of inventive faculty.
Fig. 1 schematically illustrates a schematic diagram of a distributed log data processing method in an exemplary embodiment of the present disclosure;
FIG. 2 is a schematic diagram that schematically illustrates a log data query method, in an exemplary embodiment of the disclosure;
FIG. 3 is a schematic diagram that schematically illustrates another distributed log data processing method in an exemplary embodiment of the present disclosure;
FIG. 4 schematically illustrates a composition diagram of a distributed log data processing system in an exemplary embodiment of the disclosure;
FIG. 5 schematically illustrates a schematic diagram of a distributed log data processing apparatus in an exemplary embodiment of the disclosure;
FIG. 6 schematically illustrates a schematic diagram of a distributed log data processing apparatus in an exemplary embodiment of the disclosure;
fig. 7 schematically illustrates another schematic diagram of a distributed log data processing apparatus in an exemplary embodiment of the present disclosure;
fig. 8 schematically illustrates still another schematic diagram of a distributed log data processing apparatus in an exemplary embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus their repetitive description will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in the form of software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.
The present exemplary embodiment first provides a distributed log data processing method, which may be applied to checking and saving log data of an application program based on a micro service cluster. Referring to fig. 1, the distributed log data processing method described above may include the steps of:
s101, receiving a compliance check log of a micro service container; wherein the compliance audit log includes a service data identifier;
s102, merging the related compliance check logs according to the service data identification to obtain a merged log;
s103, checking the merged log according to a first checking rule for checking the main logic of the service;
and S104, when the checking result of the merged log is abnormal, sending a log extracting instruction to the corresponding micro-service container according to the service data identifier so as to conveniently obtain a complete micro-service log corresponding to the compliance check log.
On one hand, the distributed log data processing method provided by the present exemplary embodiment receives and merges at least one compliance check log from a plurality of micro service containers to obtain a merged log, and inspects the merged log according to a preset rule, thereby implementing accurate positioning and fast troubleshooting on abnormal data. On the other hand, when the merged log is judged to be abnormal, the complete micro-service log corresponding to the micro-service container is actively acquired, so that log data are saved, subsequent data processing is facilitated, and the log data searching difficulty is reduced.
Hereinafter, each step of the distributed log data processing method in the present exemplary embodiment will be described in more detail with reference to the drawings and the examples.
Step S101, receiving a compliance check log of a micro service container; wherein the compliance audit log includes a service data identifier.
In this example embodiment, for an application program that employs distributed and micro-service clustering, each micro-service needs to be deployed in a container, for example, a micro-service cluster based on a Docker container, and logs of the application program are stored in corresponding Docker containers, respectively.
A service request from a user may go through n micro service containers, each of which performs a task. For example, in order to identify a hotel order request generated by a user, the service request may be first configured with a service request identifier for marking the order request of the user, and the service request identifier may be a service request number.
If the hotel order request of the user includes four links, for example: the verification link, the business 1 link for verifying the personnel living in, the MQ message queue link for message transmission and the business 2 link for verifying other data, the four links correspond to four micro-services, and each micro-service correspondingly executes own task and corresponding task data. For example, the data such as input parameters, output parameters, execution results (for example, an abnormal event occurs and an execution flow or a completed task is outputted), execution time, and IP address of the corresponding micro service container of the task executed by each micro service container.
In addition, when each micro service executes a task, in order to distinguish a plurality of micro service containers through which a user service request passes, a micro service calling identifier can be configured for each micro service according to a calling sequence and a calling process of the micro service. According to the service request identifier and the micro-service call identifier, the micro-service container and the specific execution sequence related to the service request can be determined.
For each micro service container, a second detection rule can be established according to the information of the executed task, the task flow and the like, and is used for checking the compliance, the service logic and the specific parameters of the task executed by each micro service container, and generating a compliance check log according to the check result and sending the compliance check log to the central log server. Moreover, the second detection rule can be adjusted according to the specific content and parameters of the task executed by each microservice, for example, different compliance check rules can be used in the verification link and the business 1 link. The present disclosure does not specifically limit the specific content of the second detection rule.
Specifically, the compliance audit log may include compliance information for normal execution of the task, or non-compliance information for exiting the execution flow due to an exception to the task. Specifically, when the micro service container normally executes the task and no abnormality occurs, the micro service container may perform compliance detection using the second detection rule and generate a corresponding compliance detection log. For example, the compliance detection log may include data such as input parameters and output parameters corresponding to the micro-service, and an IP address of a corresponding micro-service container.
If the micro service is abnormal and ends the task when executing the task, the corresponding compliance check log may include input parameters, output parameters and execution results, and may further include data such as a reason and related parameters of abnormal end, execution time and an IP address of a corresponding micro service container.
In addition, in order to facilitate to distinguish each compliance check log, a service data identifier may be configured for each compliance check log, and the service data identifier may include the service request identifier and the micro-service invocation identifier. In addition, the second detection rule can be executed in the form of a plug-in, and each running task of the microserver is monitored.
By arranging the central log server to only receive the compliance check logs containing important information, the data transmission quantity can be effectively reduced, the data pressure of the central log server is reduced, and network congestion is avoided.
And step S102, merging the related compliance check logs according to the service data identification to obtain a merged log.
In this exemplary embodiment, for the central log server, after receiving the multiple compliance check logs from the multiple microservices, the compliance check logs may be merged according to the service data identifier, so as to obtain a complete merged log of each link of the user service request. Specifically, the method comprises the following steps:
step S1021, at least one compliance check log is obtained according to the service request identifier;
and step S1022, merging the compliance check logs according to the micro-service calling identifier and the calling sequence.
Step S103, checking the merged log according to a first checking rule for checking the service master logic.
In this exemplary embodiment, a first detection rule may be predefined in the central log server, and a global compliance check may be performed on the merged log by using the first detection rule, so as to check log information of each micro service process that the user service request passes through. For example, data such as a plurality of input parameters and output parameters related to the merged log after the merging log is connected in series can be checked, and then the abnormal data in the merged log can be checked and verified.
And step S104, when the checking result of the merged log is abnormal, sending a log extracting instruction to the corresponding micro-service container according to the service data identifier so as to obtain a complete micro-service log corresponding to the compliance check log.
In this exemplary embodiment, specifically, the step S104 may include:
step S1041, when the checking result of the merged log is abnormal, generating a log extracting instruction;
step S1042, adding the log extraction instruction into the current period log extraction instruction list so as to send the log extraction instruction to the corresponding micro service container at the preset period time.
In the present exemplary embodiment, the above-described checking result presence abnormality may be a log data presence abnormality or a logical presence abnormality of data. For example, the user service request passes through four micro-services, and for each micro-service container, in this embodiment, if all the four micro-services pass the compliance check, but when the central log server performs the global compliance check on the merged log by using the first check rule, some global-based exceptions may be checked. For example, the input parameters and the output parameters of each micro service container included in the merge log are globally checked by a first check rule. At this time, if the abnormal data or logic abnormality is found through the inspection, a log extraction instruction can be sent to the corresponding micro service container, so that the micro service container uploads the complete log information to the central log server.
In another embodiment, if the task of the first micro service container is executed normally, but the second micro service container is abnormal and causes the execution flow to exit, the central log server may find that the merged log is incomplete and the abnormal data of the second micro service container when checking the merged log. In this case, it may be determined whether the abnormal data of the second micro service container is fusible and ignorable according to the first detection rule, and if the abnormal data of the second micro service container is ignorable globally, that is, only the abnormal data exists but the operation logic is not affected, it may be determined that the task of the second micro service is normal and no other processing is performed on the task.
In another embodiment, if the first and second microservices are normally executed, but an exception occurs in the third microservice and causes the execution flow to exit, the central log server may find that the merged log is incomplete when checking the merged log, and find that the exception occurs in the third microservice. When the first detection rule judges that the abnormal data of the third micro service container is not merged and is not negligible, namely when data abnormality exists and business logic abnormality is possibly caused, a log extraction instruction can be sent to the third micro service container at the moment, so that complete log information of the third micro service container during execution of the task is obtained.
Alternatively, in other exemplary embodiments of the present disclosure, the log extraction instruction may also be sent to each of the first, second, and third microservice containers, so as to obtain complete log information of the three microservice containers when the task is executed. And the abnormal data can be checked and verified accurately.
Of course, in other exemplary embodiments of the present disclosure, the complete microservice log may include a log of operations of the microservice container when executing the task, and may also include a log of compliance checks of the microservice container. In addition, the first check rule may verify the integrity of the merged log, the validity of each parameter in the merged log, and whether there is an abnormality, and may specifically set different detection rules according to the content of different service requests and different data types, and the content of the first check rule is not specifically limited by the present disclosure.
Furthermore, the periodic sending of the log extraction instruction can be set, for example, the sending period of the instruction is set to be 5 minutes or 10 minutes, and after the micro service container receives the instruction, the micro service container can search local log information and send the local log information to the central log server, so that the central log server can acquire a plurality of log data from the micro service container at one time. By respectively sending an instruction which needs to upload local running log information to each micro-service container according to a certain period, the data transmission times can be reduced, the data pressure of the micro-service containers and the central log server is reduced, and the log data can be conveniently managed.
Based on the above, in other exemplary embodiments of the present disclosure, after receiving the complete microservice log uploaded by each microservice container, the central log server may merge the operation log and/or the compliance check log of the microservice container in sequence according to the microservice call identifier, and further store the complete log data corresponding to the abnormal user service request in the central log server.
Based on the above, in the present exemplary embodiment, the method described above may further include:
step S105, when it is checked that there is no abnormality in the merged log data, deleting the merged log and/or the compliance check log corresponding to the merged log.
By setting the central log server, when the merged log is checked to have abnormality, the complete micro-service log is extracted from the micro-service container, and the compliance check log and the merged log are deleted when the merged log is checked to have no abnormality, the important log data can be effectively and timely stored, and the storage quantity of the log data can be reduced.
In addition, in other exemplary embodiments of the present disclosure, an interactive interface may be provided for the central log server, and the user may input log information to be queried or verified on the interactive interface. Referring to fig. 2, the method described above may further include:
step S201, receiving a log data query request;
and S202, extracting a corresponding complete micro service log according to the log data query request.
In the exemplary embodiment, the user may submit the relevant information of the service request on the interactive interface, for example: user ID, order number, time, product information, etc. And searching the uploaded log or the merged complete log data locally at the central log server through the related information of the service request. And further, the number of log retrieval is reduced, log information does not need to be inquired for each micro-service container, and unified retrieval of log data and quick troubleshooting of abnormal data are facilitated.
For example, when a technician receives an online troubleshooting task, the technician can search corresponding log data in a central log server according to related information submitted by a user, so as to obtain a merged log corresponding to a user service request. Of course, the operation log or/and the compliance check log of the micro service container corresponding to the user service request may also be found. Thereby facilitating troubleshooting of anomalous or erroneous data in the log data.
Further, another distributed log data processing method is also provided in this example embodiment, and as shown in fig. 3, the method includes:
step S301, checking the current task according to a second detection rule for checking the business logic to generate a current compliance check log;
step S302, generating a service data identifier in the current compliance check log to obtain the compliance check log;
step S303, sending the compliance check log to a central log server so that the central log server checks the compliance check log.
In this exemplary embodiment, for the micro service container, when executing a task corresponding to the user service request, local compliance check may be performed on the current task according to a preset second detection rule, and a service data identifier is configured for the generated current compliance check log, so as to generate a compliance check log of the micro service container on the current task. And sends the compliance audit log to a central log server.
For example, the compliance check log may include compliance information of the task during normal execution, such as input parameters, output parameters, and IP addresses of corresponding micro service containers of the task. Alternatively, the compliance check log may further include non-compliance information generated by the task exiting the execution flow due to the exception, for example, data such as input parameters, output parameters, execution results, execution time, related exception data, and IP address of the corresponding micro service container may be included. In addition, the second detection rule can also be executed in the form of plug-in the micro service container, and each running task of the micro service is monitored.
Further, in this exemplary embodiment, in the above step S302, adding a service data identifier to the current compliance audit log to obtain the compliance audit log further includes:
step S3021, extracting current log data from the current compliance check log according to a preset field;
step S3022, generating the service data identifier and adding the service data identifier to the current log data to obtain the compliance check log.
For the compliance check log, fixed parameters or important information can be extracted according to preset fields or preset rules to generate an abstract, so that the information of the compliance check log is as little and effective as possible, all local log information is prevented from being submitted to a central log server, and network congestion is further avoided.
Based on the above, in other exemplary embodiments of the present disclosure, the above method may further include: and if a log extraction instruction corresponding to the compliance check log is received, extracting a complete microservice log corresponding to the compliance check log according to the log extraction instruction.
Wherein the complete microservice log comprises: a log of operations of the microservice container and a log of compliance detection of the microservice container.
For the micro service container, when receiving a log extraction instruction from the central log server, the micro service container can locally search the running log and the compliance check log of the corresponding task according to the service data identifier contained in the instruction information, and send all log information to the central log server. The central log server can combine the running logs according to the service request identifier and the micro-service calling identifier, thereby recovering the internal operation of the micro-service experienced by the user service request and generating related abnormal information.
The distributed log data processing method provided by the present disclosure, by making the micro service containers perform compliance check on the tasks executed by the micro service containers, generates a compliance check log and sends the compliance check log to a central log server; the central log server can combine a plurality of associated compliance check logs according to the service data identification and the calling sequence of the microservice, and then check the combined logs according to a preset detection rule. And only when the central log server detects that the merged log is abnormal, the corresponding complete data is acquired from the micro service containers, so that the central log server does not need to receive all log information of each micro service container, the data transmission pressure between the micro service containers and the central log server and the storage pressure of the central log server are reduced, all logs under the abnormal condition can be uploaded to the central log server to be stored, and the abnormal data can be rapidly checked and acquired conveniently.
It is to be noted that the above-mentioned figures are only schematic illustrations of the processes involved in the method according to an exemplary embodiment of the invention, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
Further, in the present exemplary embodiment, there is also provided a distributed log data processing system, as shown with reference to fig. 4, including: a central log server, and at least one microservice container;
the micro service container is used for checking a current task according to a second detection rule for checking business logic to generate a compliance detection log so as to send the compliance detection log to the central log server; wherein the compliance detection log comprises a service data identifier;
the central log server is used for merging the compliance detection logs according to the service data identification to obtain merged logs, and checking the merged logs according to a first check rule for checking service main logic to obtain abnormal data.
The specific details of each server in the distributed log data processing system have been described in detail in the corresponding distributed log data processing method, and therefore are not described herein again.
Further, referring to fig. 5, in the present exemplary embodiment, there is also provided a distributed log data processing apparatus 50, including: a log data receiving module 501, a log data merging module 502, a log data checking module 503, and a complete log data extracting module 504. Wherein:
the log data receiving module 501 may be configured to receive a compliance check log of a micro service container; wherein the compliance audit log includes a service data identifier.
The log data merging module 502 may be configured to merge the associated compliance check logs according to the service data identifier to obtain a merged log.
The log data checking module 503 may be configured to check the merged log according to a first checking rule for checking the business master logic.
The complete log data extraction module 504 may be configured to, when the inspection result of the merged log is abnormal, send a log extraction instruction to the corresponding micro service container according to the service data identifier, so as to obtain a complete micro service log corresponding to the compliance check log.
Further, referring to fig. 6, in the present exemplary embodiment, there is also provided a distributed log data processing apparatus 60, including: a log data generating module 601, an identification loading module 602 and a log data sending module 603. Wherein the content of the first and second substances,
the log data generating module 601 may be configured to verify the current task according to a second detection rule for verifying the business logic to generate a current compliance check log.
The identifier loading module 602 may be configured to add a service data identifier to the current compliance check log to obtain a compliance check log.
The log data sending module 603 may be configured to send the compliance check log to a central log server so that the central log server checks the compliance check log.
The specific details of each module in the distributed log data processing apparatus have been described in detail in the corresponding distributed log data processing method, and therefore are not described herein again.
It should be noted that although in the above detailed description several modules or units of the device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit, according to embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into embodiments by a plurality of modules or units.
In an exemplary embodiment of the present disclosure, there is also provided a computer system capable of implementing the above method.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
A computer system 600 according to this embodiment of the invention is described below with reference to fig. 6. The computer system 600 shown in fig. 6 is only an example and should not bring any limitations to the function and scope of use of the embodiments of the present invention.
As shown in FIG. 6, computer system 600 is in the form of a general purpose computing device. The components of computer system 600 may include, but are not limited to: the at least one processing unit 610, the at least one memory unit 620, and a bus 630 that couples the various system components including the memory unit 620 and the processing unit 610.
Wherein the storage unit stores program code that is executable by the processing unit 610 to cause the processing unit 610 to perform steps according to various exemplary embodiments of the present invention as described in the above section "exemplary methods" of the present specification. For example, the processing unit 610 may perform step S101 as shown in fig. 1: receiving a compliance audit log of a microservice container; wherein the compliance audit log includes a service data identifier; s102: merging the associated compliance check logs according to the service data identifier to obtain a merged log; s103: checking the merged log according to a first checking rule for checking the service main logic; s104: when the checking result of the merged log is abnormal, a log extracting instruction is sent to the corresponding micro-service container according to the service data identifier so as to obtain a complete micro-service log corresponding to the compliance check log; or to perform the steps as shown in fig. 3: step S301: checking the current task according to a second detection rule for checking the business logic to generate a current compliance check log; step S302: generating a service data identifier in the current compliance check log to obtain the compliance check log; step S303: and sending the compliance check log to a central log server so that the central log server can check the compliance check log.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The computer system 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the computer system 600, and/or with any devices (e.g., router, modem, etc.) that enable the computer system 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Moreover, computer system 600 may also communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network such as the Internet) via network adapter 660. As shown, network adapter 660 communicates with the other modules of computer system 600 via bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with computer system 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein may be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.
In an exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium having stored thereon a program product capable of implementing the above-described method of the present specification. In some possible embodiments, aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention described in the above section "exemplary methods" of the present description, when said program product is run on the terminal device.
Referring to fig. 7, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable signal medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
Furthermore, the above-described figures are merely schematic illustrations of processes involved in methods according to exemplary embodiments of the invention, and are not intended to be limiting. It will be readily understood that the processes shown in the above figures are not intended to indicate or limit the chronological order of the processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, e.g., in multiple modules.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It will be understood that the present disclosure is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is to be limited only by the terms of the appended claims.
Claims (15)
1. A distributed log data processing method, comprising:
receiving a compliance audit log of a microservice container; wherein the compliance audit log includes a service data identifier;
merging the associated compliance check logs according to the service data identifier to obtain a merged log;
checking the merged log according to a first checking rule for checking the service main logic;
and when the checking result of the merged log is abnormal, sending a log extracting instruction to the corresponding micro-service container according to the service data identifier so as to obtain a complete micro-service log corresponding to the compliance check log.
2. The method of claim 1, further comprising:
and when the merged log data is checked to have no abnormity, deleting the merged log and/or the compliance check log corresponding to the merged log.
3. The method of claim 1, wherein the service data identification comprises: the micro-service container calling system comprises a service request identifier used for marking service requests and a micro-service calling identifier used for identifying the calling sequence of the micro-service container.
4. The method of claim 3, wherein the merging the associated compliance check logs according to the service data identifier to obtain a merged log comprises:
acquiring at least one compliance check log according to the service request identifier;
and merging the compliance check logs according to the micro-service calling identifier according to a calling sequence.
5. The method according to claim 3, wherein when it is determined that the merged log has abnormal data, sending a log extraction instruction to the corresponding microservice container according to the service data identifier so as to obtain a complete microservice log corresponding to the compliance check log comprises:
when judging that the merged log has abnormal data, generating a log extraction instruction;
and adding the log extraction instruction into a current period log extraction instruction list so as to send the log extraction instruction to the corresponding micro service container at the moment of a preset period.
6. The method of claim 5, wherein the complete microservice log comprises:
a log of runs of the microservice container and a log of compliance checks of the microservice container.
7. The method of claim 1, wherein an interactive interface is provided, the method further comprising:
receiving a log data query request;
and extracting a corresponding complete micro service log according to the log data query request.
8. A distributed log data processing method, comprising:
checking the current task according to a second detection rule for checking the business logic to generate a current compliance check log;
generating a service data identifier in the current compliance check log to obtain the compliance check log;
and sending the compliance check log to a central log server so that the central log server can check the compliance check log.
9. The method of claim 8, wherein adding the service data identifier to the current compliance audit log to obtain the compliance audit log comprises:
extracting current log data from the current compliance check log according to a preset field;
and generating the service data identifier and adding the service data identifier to the current log data to obtain the compliance check log.
10. The method of claim 8, further comprising:
if a log extraction instruction corresponding to the compliance check log is received, extracting a complete micro service log corresponding to the compliance check log according to the log extraction instruction;
wherein the complete microservice log comprises: a log of runs of microservice containers and a log of compliance checks of the microservice containers.
11. A distributed log data processing system, comprising: a central log server, and at least one microservice container;
wherein the central log server is configured to perform the method of any of claims 1 to 7;
the microservice container is for performing the method of any of claims 8 to 10.
12. A distributed log data processing apparatus, comprising:
the log data receiving module is used for receiving a compliance check log of the microservice container; wherein the compliance audit log includes a service data identifier;
the log data merging module is used for merging the associated compliance check logs according to the service data identification to obtain a merged log;
the log data inspection module is used for inspecting the merged log according to a first inspection rule for inspecting the service main logic;
and the complete log data extraction module is used for sending a log extraction instruction to the corresponding micro-service container according to the service data identifier when the checking result of the combined log is abnormal so as to obtain the complete micro-service log corresponding to the compliance check log.
13. A distributed log data processing apparatus, comprising:
the log data generation module is used for checking the current task according to a second detection rule for checking the business logic to generate a current compliance check log;
the identification loading module is used for adding a service data identification to the current compliance check log so as to obtain a compliance check log;
and the log data sending module is used for sending the compliance check log to a central log server so as to be convenient for the central log server to check the compliance check log.
14. A storage medium having stored thereon a computer program which, when executed by a processor, implements a distributed log data processing method according to any one of claims 1 to 7 or claims 8 to 10.
15. A computer system, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the distributed log data processing method of any of claims 1 to 7 or claims 8 to 10 via execution of the executable instructions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810848162.9A CN110851471A (en) | 2018-07-27 | 2018-07-27 | Distributed log data processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810848162.9A CN110851471A (en) | 2018-07-27 | 2018-07-27 | Distributed log data processing method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110851471A true CN110851471A (en) | 2020-02-28 |
Family
ID=69594896
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810848162.9A Pending CN110851471A (en) | 2018-07-27 | 2018-07-27 | Distributed log data processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110851471A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111786828A (en) * | 2020-06-29 | 2020-10-16 | 百度在线网络技术（北京）有限公司 | Log processing method executed in network device and network device |
| CN111813498A (en) * | 2020-07-02 | 2020-10-23 | 深圳市国电科技通信有限公司 | Terminal container monitoring method, monitoring device, storage medium and processor |
| CN111858527A (en) * | 2020-06-22 | 2020-10-30 | 清华大学深圳国际研究生院 | Log data modeling analysis method and computer readable storage medium |
| CN112182506A (en) * | 2020-09-16 | 2021-01-05 | 支付宝(杭州)信息技术有限公司 | Data compliance detection method, device and equipment |
| WO2021248201A1 (en) * | 2020-06-11 | 2021-12-16 | Commonwealth Scientific And Industrial Research Organisation | "log data compliance" |
| CN114281280A (en) * | 2021-12-31 | 2022-04-05 | 中企云链（北京）金融信息服务有限公司 | Log processing method and device, storage medium and electronic device |
-
2018
- 2018-07-27 CN CN201810848162.9A patent/CN110851471A/en active Pending
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021248201A1 (en) * | 2020-06-11 | 2021-12-16 | Commonwealth Scientific And Industrial Research Organisation | "log data compliance" |
| CN111858527A (en) * | 2020-06-22 | 2020-10-30 | 清华大学深圳国际研究生院 | Log data modeling analysis method and computer readable storage medium |
| CN111786828A (en) * | 2020-06-29 | 2020-10-16 | 百度在线网络技术（北京）有限公司 | Log processing method executed in network device and network device |
| CN111813498A (en) * | 2020-07-02 | 2020-10-23 | 深圳市国电科技通信有限公司 | Terminal container monitoring method, monitoring device, storage medium and processor |
| CN112182506A (en) * | 2020-09-16 | 2021-01-05 | 支付宝(杭州)信息技术有限公司 | Data compliance detection method, device and equipment |
| CN114281280A (en) * | 2021-12-31 | 2022-04-05 | 中企云链（北京）金融信息服务有限公司 | Log processing method and device, storage medium and electronic device |
| CN114281280B (en) * | 2021-12-31 | 2023-08-25 | 中企云链（北京）金融信息服务有限公司 | Log processing method and device, storage medium and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11449379B2 (en) | Root cause and predictive analyses for technical issues of a computing environment | |
| CN110851471A (en) | Distributed log data processing method, device and system | |
| US9612936B2 (en) | Correlation of source code with system dump information | |
| US20150347212A1 (en) | Error classification in a computing system | |
| Syer et al. | Continuous validation of performance test workloads | |
| CN110851324B (en) | Log-based routing inspection processing method and device, electronic equipment and storage medium | |
| CN109508295B (en) | Block chain consensus algorithm testing method and device, calculating device and storage medium | |
| WO2021097824A1 (en) | Code quality and defect analysis method, server and storage medium | |
| CN113014445B (en) | Operation and maintenance method, device and platform for server and electronic equipment | |
| CN109542781B (en) | Block chain consensus algorithm testing method and device, calculating device and storage medium | |
| US9612827B2 (en) | Automatically complete a specific software task using hidden tags | |
| US10795793B1 (en) | Method and system for simulating system failures using domain-specific language constructs | |
| CN112817853A (en) | Automatic test method, system and electronic equipment | |
| CN112363938A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN109299124B (en) | Method and apparatus for updating a model | |
| US8380729B2 (en) | Systems and methods for first data capture through generic message monitoring | |
| CN110875832B (en) | Abnormal service monitoring method, device and system and computer readable storage medium | |
| US9354962B1 (en) | Memory dump file collection and analysis using analysis server and cloud knowledge base | |
| CN110618943B (en) | Security service test method and device, electronic equipment and readable storage medium | |
| CN113918525A (en) | Data exchange scheduling method, system, electronic device, medium, and program product | |
| US10796264B2 (en) | Risk assessment in online collaborative environments | |
| CN114168108A (en) | Code merging method and device, electronic equipment and computer readable storage medium | |
| CN110362464B (en) | Software analysis method and equipment | |
| CN113535568B (en) | Verification method, device, equipment and medium for application deployment version | |
| US9436523B1 (en) | Holistic non-invasive evaluation of an asynchronous distributed software process |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |